Story #2525
closed
Failover events should be explicitly logged as such when they occur
Status:
CLOSED - CURRENTRELEASE
Description
Problem¶
pulp_resource_manager and pulp_celerybeat both use hot-spare processes for high availability. The current behavior logs when a worker go missing, when a new ones comes online, and when the lock is acquired.
Solution¶
Whenever a pulp_resource_manager or pulp_celerybeat instance is a hot spare (could not require the lock) and then becomes the primary (acquires the lock), logs should notify the user that failover has occurred. Specifically at WARNING level, it should log the following:
Failover occured: xxxxxx is now the primary
Additionally, the "lock acquired" log statement should be switched to DEBUG since the WARNING statement will be shown to the user at that same moment.
Do you think this should be at the WARNING level?
- Status changed from NEW to ASSIGNED
- Status changed from ASSIGNED to NEW
- Assignee deleted (
dalley)
Yes, I think that would be appropriate.
- Description updated (diff)
- Sprint Candidate changed from No to Yes
I've rewritten this some. Now it suggests that these WARNING level logs will replace the "lock acquired" logs. I think that is good because that is effectively what that log statement means.
Is ^ a good idea?
Would it be better to just add the WARNING level statement, and convert the "lock acquired" ones to DEBUG?
I think the latter suggestion would be good - keep the lock acquired logs but make them DEBUG level
- Description updated (diff)
- Description updated (diff)
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
- Groomed changed from No to Yes
- Status changed from ASSIGNED to POST
- Sprint/Milestone set to 32
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Platform Release set to 2.12.0
- Status changed from MODIFIED to 5
- Status changed from 5 to CLOSED - CURRENTRELEASE
- Sprint changed from Sprint 16 to Sprint 14
- Sprint/Milestone deleted (
32)
Also available in: Atom
PDF
Explictly log failovers
Failover events are now logged as such at the WARNING level
closes #2525