Pulp

Current issues¶

• Pulp cannot be Highly Available due to RQ not supporting Redis in a clustered mode.
• Pulp has an architectural throughput of ~ 2.6 tasks / sec.
• Architecturally synchronizing data between Redis and Postgresql is practically difficult and causes reliability issues for Pulp users.

Goals¶

1. Eliminate the resource manager. This will address the architectural throughput issue while also simplifying Pulp deployments.
2. Introduce a new worker type that is not RQ based and instead works entirely out of the Task table in Postgresql. This will resolve the high availability issue and the data synchronization data.

Plan for introducing this to users¶

Both the new-style and the old, RQ-style workers will be available in the codebase at the same time. This will allow users to try the new-style workers, but also be able to go back to the old style if any issues come up.

The new-style workers will be introduced in the following phases over a variety of releases:

1. Introduce the new-style workers and leave the old-style workers as the default. A new setting named USE_NEW_WORKER_TYPE will allow users to opt-in to the new-style.
2. Add support to the installer, containers (s6), and operator to allow users to also opt-in
3. In a specific release, change the default of USE_NEW_WORKER_TYPE=False to USE_NEW_WORKER_TYPE=True. This will still keep the old-style workers around so users can opt-out back to the old style if they need to.
4. Remove the old-style worker and its codepaths entirely. Drop RQ as a dependency.