Project

Profile

Help

Story #1060

As a user I can run Pulp with multiple celeryBeat instances

Added by sbhawsin over 5 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.8.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Quarter:

Description

pulp_celerybeat currently can only be run in one process per deployment, and it is one of the single points of failure in Pulp. Multiple instances of pulp_celerybeat should be able to run concurrently.

Deliverables:

1) Modify Pulp to be able to run multiple pulp_celerybeat process concurrently
2) Ensure that Pulp scheduled calls occur only once even with multiple pulp_celerybeat processes running
3) Ensure that even if pulp_celerybeat processes start being killed, that scheduled calls still dispatch only once.
4) Document changes
5) Add a release note
6) Verify that the failure watcher code is safe to run concurrently
7) Verify that the EventMonitor code is safe to run concurrently
8) Verify that the WorkerTimeoutMonitor code is safe to run concurrently


Related issues

Blocks Pulp - Issue #1113: If an instance of pulp_celerybeat dies unexpectedly, Pulp incorrectly tries to "cancel all tasks in its queue"CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision df6ad285 View on GitHub
Added by sbhawsin over 5 years ago

Story 1060: Pulp with multiple celerybeat instances

https://pulp/plan.io/issues/1060

closes #1060

Revision df6ad285 View on GitHub
Added by sbhawsin over 5 years ago

Story 1060: Pulp with multiple celerybeat instances

https://pulp/plan.io/issues/1060

closes #1060

History

#1 Updated by sbhawsin over 5 years ago

sbhawsin wrote:

Celerybeat is one of the single point of failure in pulp. We should be able to make pulp highly available by making celerybeat highly available.

Deliverables:
1) Able to run multiple celerybeat process
2) By killing one of the process, pulp should not fail to work

Design Doc: https://fedorahosted.org/pulp/wiki/PulpHACelerybeat

#2 Updated by bmbouter over 5 years ago

  • Subject changed from To be able to run pulp with multiple celeryBeat instances and make it HA to As a user I can run Pulp with multiple celeryBeat instances
  • Description updated (diff)

#3 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#4 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#5 Updated by mhrivnak over 5 years ago

  • Description updated (diff)
  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#6 Updated by ipanova@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST

#7 Updated by bmbouter over 5 years ago

  • Blocks Issue #1113: If an instance of pulp_celerybeat dies unexpectedly, Pulp incorrectly tries to "cancel all tasks in its queue" added

#8 Updated by sbhawsin over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#9 Updated by bmbouter over 5 years ago

QE, you should use a clustered setup to run pulp_celerybeat on two machines. Then configure some scheduled operations and ensure that the expected number of operations occur. If you kill -9 one of the pulp_celerybeat and leave the other running everything should still work including manual sync+publish operations, and scheduled operations. Note, a sync may take up to 270 seconds to failover after killing a pulp_celerybeat, but it is still expected to recover.

#10 Updated by mhrivnak almost 5 years ago

  • Platform Release changed from master to 2.8.0

#11 Updated by dkliban@redhat.com almost 5 years ago

  • Status changed from MODIFIED to 5

#12 Updated by dkliban@redhat.com over 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#13 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF