Project

Profile

Help

Story #3707

closed

As a user, I can run multiple resource_managers for high availability

Added by bmbouter over 6 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

With the switch to RQ, the resource_manager became not-highly available. If you run two of them, bad things will happen.

We should make the resource_manager highly available again. there are two implementation options:

1. Use TaskLock again

Before the RQ transition we were using TaskLock: https://github.com/pulp/pulp/blob/968766e975c2eca00169470b8a028ec28c2a9274/pulpcore/pulpcore/app/models/task.py#L250-L280

This would be a PostgreSQL based table that only serves to elect a single master by one person writing the unique lock record.

2. Built something Redis based

This would be more something to contribute upstream to RQ.

3. Find something that we can wrap around the resource manager to provide a singleton function.

Actions #1

Updated by bmbouter over 6 years ago

  • Sprint Candidate changed from No to Yes

This will fix our high availability narrative so Pulp will have 0 single points of failure. It's a small amount of work for a big benefit. As such, I'm nominating for the next sprint.

Actions #2

Updated by bmbouter over 6 years ago

  • Sprint Candidate changed from Yes to No

After talking with dalley, we're going to do this work later relative to other higher priority work. I'm removing the sprint candidate flag.

Actions #3

Updated by bmbouter over 6 years ago

  • Description updated (diff)

Updating to reflect that TaskLock is being removed from the codebase.

Actions #4

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions #5

Updated by bmbouter over 5 years ago

  • Description updated (diff)
  • Sprint/Milestone set to 3.0.0

This was expressed as desirable for Katello's usage for the project itself (not Katello end users). So it's not a P1-P4, but it is important for them to adopt Pulp3 (and probably many others).

Actions #6

Updated by bmbouter about 5 years ago

  • Sprint/Milestone deleted (3.0.0)
Actions #7

Updated by bmbouter almost 5 years ago

I just thought of an even easier option. We could make the worker name require it to be specifically "resource-manager". RQ has a behavior that if another worker already has that same name the worker cannot start. Then systemd would potentially be configured to try to restart anyway and this would allow you to have a N other hot-spares with automatic failover.

Actions #8

Updated by bmbouter almost 5 years ago

  • Sprint/Milestone set to 3.0.0

We can accomplish this very easily. I will post a POC on Monday.

Actions #9

Updated by bmbouter almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

Added by bmbouter almost 5 years ago

Revision 2047c4e7 | View on GitHub

Change resource manager name disinclude hostname

This is a backwards compatible change, but it will be required once pulpcore requires the name to be exactly resource-manager.

https://pulp.plan.io/issues/3707 re #3707

Added by bmbouter almost 5 years ago

Revision 2047c4e7 | View on GitHub

Change resource manager name disinclude hostname

This is a backwards compatible change, but it will be required once pulpcore requires the name to be exactly resource-manager.

https://pulp.plan.io/issues/3707 re #3707

Actions #10

Updated by bmbouter almost 5 years ago

  • Status changed from ASSIGNED to POST

Added by bmbouter almost 5 years ago

Revision d56a6175 | View on GitHub

Allow multiple resource-managers to be started

The resource manager name must exctly equal resource-manager and multiple of them can be started at once safely.

https://pulp.plan.io/issues/3707 closes #3707

Actions #11

Updated by bmbouter almost 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Added by bmbouter almost 5 years ago

Revision 97c26e40 | View on GitHub

Allow multiple resource-managers to be started

The resource manager name must exctly equal resource-manager and multiple of them can be started at once safely.

https://pulp.plan.io/issues/3707 closes #3707

(cherry picked from commit d56a61755350f3eeb32975f2d361ebb78f71c667)

Actions #12

Updated by bmbouter almost 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF