Task #8495
Updated by bmbouter over 3 years ago
## Current issues
* Pulp cannot be Highly Available due to [RQ not supporting Redis in a clustered mode](https://github.com/rq/rq/issues/862).
* Pulp has [an architectural throughput of ~ 2.6 tasks / sec](https://hackmd.io/DV633ocwTnShsI8LdajshA).
* Architecturally synchronizing data between Redis and Postgresql is practically difficult and causes reliability issues for Pulp users.
## Goals
1. Eliminate the resource manager. This will address the architectural throughput issue while also simplifying Pulp deployments.
2. Introduce a new worker type that is not RQ based and instead works entirely out of the Task table in Postgresql. This will resolve the high availability issue and the data synchronization data.
## Plan for introducing this to users
Both the new-style and the old, RQ-style workers will be available in the codebase at the same time. This will allow users to try the new-style workers, but also be able to go back to the old style if any issues come up.
The new-style workers will be introduced in the following phases over a variety of releases:
1. Introduce the new-style workers and leave the old-style workers as the default. A new setting named `USE_NEW_WORKER_TYPE` will allow users to opt-in to the new-style.
2. Add support to the installer, containers (s6), and operator to allow users to also opt-in
3. In a specific release, change the default of `USE_NEW_WORKER_TYPE=False` to `USE_NEW_WORKER_TYPE=True`. This will still keep the old-style workers around so users can opt-out back to the old style if they need to.
4. Remove the old-style worker and its codepaths entirely. Drop RQ as a dependency.
.