Issue #2516
closed
worker heartbeat log statement mishandles timezone
Description
From bugzilla:
Dec 22 14:03:55 pmoravec-sat62-rhel6 pulp: pulp.server.async.worker_watcher:DEBUG: 'worker-heartbeat' sent at time 2016-12-22 14:03:55.439300 from reserved_resource_worker-3@pmoravec-sat62-rhel6.gsslab.brq.redhat.com, received at time: 2016-12-22 13:03:55.450517
Saying that:
- worker sent heartbeat at 14:03
- the heartbeat was received at 13:03
The machine was set to GTM+1.
The code is here:
https://github.com/pulp/pulp/blob/pulp-2.11.0-1/server/pulp/server/async/worker_watcher.py#L27
We generate our own timestamps and only use the worker name from the heartbeat. Instead of logging the heartbeat we receive, we should create our own heartbeat log line with the worker name and timestamp that we are using (the one we generate).
- Sprint/Milestone set to 31
- Severity changed from 2. Medium to 1. Low
- Triaged changed from No to Yes
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
- Sprint/Milestone changed from 31 to 32
- Status changed from ASSIGNED to NEW
- Assignee deleted (
dalley)
Unassigning myself for grooming - jumped the gun a bit
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
- Status changed from ASSIGNED to POST
- Status changed from POST to MODIFIED
Manual verification as per smash issue #513
- Platform Release set to 2.13.0
- Status changed from MODIFIED to 5
- Status changed from 5 to CLOSED - CURRENTRELEASE
- Status changed from CLOSED - CURRENTRELEASE to CLOSED - WONTFIX
See discussion in the associated bugzilla. Revisited due to FailedQA, but we recognized that it is perfectly fine to remain the way it is currently, and isn't an issue.
- Status changed from CLOSED - WONTFIX to CLOSED - NOTABUG
I suppose this should actually be "NOTABUG" instead of "WONTFIX"
- Sprint changed from Sprint 16 to Sprint 14
- Sprint/Milestone deleted (
32)
Also available in: Atom
PDF
.
Workers write their own hearbeat records to database.
All workers will write their own records to the database instead of relying on pulp_celerybeat to do so for them using celery heartbeats.
This patch makes use of the Consumer blueprint that celery runs at the start time of a worker. An extra boot step has been added which sets a timer to periodically update the worker record in the database.
http://docs.celeryproject.org/en/master/userguide/extending.html https://groups.google.com/d/msg/celery-users/3fs0ocREYqw/C7U1lCAp56sJ
closes #2519 https://pulp.plan.io/issues/2519 closes #2516 https://pulp.plan.io/issues/2516