As a user I can run Pulp with multiple celeryBeat instances
pulp_celerybeat currently can only be run in one process per deployment, and it is one of the single points of failure in Pulp. Multiple instances of pulp_celerybeat should be able to run concurrently.
1) Modify Pulp to be able to run multiple pulp_celerybeat process concurrently
2) Ensure that Pulp scheduled calls occur only once even with multiple pulp_celerybeat processes running
3) Ensure that even if pulp_celerybeat processes start being killed, that scheduled calls still dispatch only once.
4) Document changes
5) Add a release note
6) Verify that the failure watcher code is safe to run concurrently
7) Verify that the EventMonitor code is safe to run concurrently
8) Verify that the WorkerTimeoutMonitor code is safe to run concurrently
Updated by sbhawsin over 8 years ago
Celerybeat is one of the single point of failure in pulp. We should be able to make pulp highly available by making celerybeat highly available.
1) Able to run multiple celerybeat process
2) By killing one of the process, pulp should not fail to work
Updated by bmbouter over 8 years ago
QE, you should use a clustered setup to run pulp_celerybeat on two machines. Then configure some scheduled operations and ensure that the expected number of operations occur. If you kill -9 one of the pulp_celerybeat and leave the other running everything should still work including manual sync+publish operations, and scheduled operations. Note, a sync may take up to 270 seconds to failover after killing a pulp_celerybeat, but it is still expected to recover.