Story #2525
closed
Failover events should be explicitly logged as such when they occur
100%
Description
Problem¶
pulp_resource_manager and pulp_celerybeat both use hot-spare processes for high availability. The current behavior logs when a worker go missing, when a new ones comes online, and when the lock is acquired.
Solution¶
Whenever a pulp_resource_manager or pulp_celerybeat instance is a hot spare (could not require the lock) and then becomes the primary (acquires the lock), logs should notify the user that failover has occurred. Specifically at WARNING level, it should log the following:
Failover occured: xxxxxx is now the primary
Additionally, the "lock acquired" log statement should be switched to DEBUG since the WARNING statement will be shown to the user at that same moment.
Updated by bmbouter over 7 years ago
Do you think this should be at the WARNING level?
Updated by dalley over 7 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
dalley)
Updated by bmbouter over 7 years ago
- Description updated (diff)
- Sprint Candidate changed from No to Yes
Updated by bmbouter over 7 years ago
I've rewritten this some. Now it suggests that these WARNING level logs will replace the "lock acquired" logs. I think that is good because that is effectively what that log statement means.
Is ^ a good idea?
Would it be better to just add the WARNING level statement, and convert the "lock acquired" ones to DEBUG?
Updated by dalley over 7 years ago
I think the latter suggestion would be good - keep the lock acquired logs but make them DEBUG level
Updated by dalley over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
- Groomed changed from No to Yes
Added by dalley over 7 years ago
Added by dalley over 7 years ago
Updated by dalley over 7 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|34c45b2d7fc48f4f6f25dc0339f87da5ce91932e.
Updated by semyers about 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Updated by bmbouter about 6 years ago
- Sprint changed from Sprint 16 to Sprint 14
.
Explictly log failovers
Failover events are now logged as such at the WARNING level
closes #2525