Story #3707
closed
As a user, I can run multiple resource_managers for high availability
100%
Description
With the switch to RQ, the resource_manager became not-highly available. If you run two of them, bad things will happen.
We should make the resource_manager highly available again. there are two implementation options:
1. Use TaskLock again¶
Before the RQ transition we were using TaskLock: https://github.com/pulp/pulp/blob/968766e975c2eca00169470b8a028ec28c2a9274/pulpcore/pulpcore/app/models/task.py#L250-L280
This would be a PostgreSQL based table that only serves to elect a single master by one person writing the unique lock record.
2. Built something Redis based¶
This would be more something to contribute upstream to RQ.
3. Find something that we can wrap around the resource manager to provide a singleton function.¶
Updated by bmbouter over 6 years ago
- Sprint Candidate changed from No to Yes
This will fix our high availability narrative so Pulp will have 0 single points of failure. It's a small amount of work for a big benefit. As such, I'm nominating for the next sprint.
Updated by bmbouter over 6 years ago
- Sprint Candidate changed from Yes to No
After talking with dalley, we're going to do this work later relative to other higher priority work. I'm removing the sprint candidate flag.
Updated by bmbouter over 6 years ago
- Description updated (diff)
Updating to reflect that TaskLock is being removed from the codebase.
Updated by bmbouter over 5 years ago
- Description updated (diff)
- Sprint/Milestone set to 3.0.0
This was expressed as desirable for Katello's usage for the project itself (not Katello end users). So it's not a P1-P4, but it is important for them to adopt Pulp3 (and probably many others).
Updated by bmbouter almost 5 years ago
I just thought of an even easier option. We could make the worker name require it to be specifically "resource-manager". RQ has a behavior that if another worker already has that same name the worker cannot start. Then systemd would potentially be configured to try to restart anyway and this would allow you to have a N other hot-spares with automatic failover.
Updated by bmbouter almost 5 years ago
- Sprint/Milestone set to 3.0.0
We can accomplish this very easily. I will post a POC on Monday.
Updated by bmbouter almost 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to bmbouter
Added by bmbouter almost 5 years ago
Added by bmbouter almost 5 years ago
Revision 2047c4e7 | View on GitHub
Change resource manager name disinclude hostname
This is a backwards compatible change, but it will be required once
pulpcore requires the name to be exactly resource-manager.
Updated by bmbouter almost 5 years ago
- Status changed from ASSIGNED to POST
PR available at: https://github.com/pulp/pulpcore/pull/427
Added by bmbouter almost 5 years ago
Revision d56a6175 | View on GitHub
Allow multiple resource-managers to be started
The resource manager name must exctly equal resource-manager and
multiple of them can be started at once safely.
Updated by bmbouter almost 5 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulpcore|d56a61755350f3eeb32975f2d361ebb78f71c667.
Added by bmbouter almost 5 years ago
Revision 97c26e40 | View on GitHub
Allow multiple resource-managers to be started
The resource manager name must exctly equal resource-manager and
multiple of them can be started at once safely.
https://pulp.plan.io/issues/3707 closes #3707
(cherry picked from commit d56a61755350f3eeb32975f2d361ebb78f71c667)
Updated by bmbouter almost 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Change resource manager name disinclude hostname
This is a backwards compatible change, but it will be required once pulpcore requires the name to be exactly
resource-manager.https://pulp.plan.io/issues/3707 re #3707