Project

Profile

Help

Issue #2516

worker heartbeat log statement mishandles timezone

Added by mhrivnak about 4 years ago. Updated almost 2 years ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
2.8.7
Platform Release:
2.13.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix, Pulp 2
Sprint:
Sprint 14
Quarter:

Description

From bugzilla:

Dec 22 14:03:55 pmoravec-sat62-rhel6 pulp: pulp.server.async.worker_watcher:DEBUG: 'worker-heartbeat' sent at time 2016-12-22 14:03:55.439300 from reserved_resource_worker-3@pmoravec-sat62-rhel6.gsslab.brq.redhat.com, received at time: 2016-12-22 13:03:55.450517

Saying that:
- worker sent heartbeat at 14:03
- the heartbeat was received at 13:03

The machine was set to GTM+1.

The code is here:

https://github.com/pulp/pulp/blob/pulp-2.11.0-1/server/pulp/server/async/worker_watcher.py#L27

Associated revisions

Revision fd19f890 View on GitHub
Added by dalley almost 4 years ago

Workers write their own hearbeat records to database.

All workers will write their own records to the database instead of relying on pulp_celerybeat to do so for them using celery heartbeats.

This patch makes use of the Consumer blueprint that celery runs at the start time of a worker. An extra boot step has been added which sets a timer to periodically update the worker record in the database.

http://docs.celeryproject.org/en/master/userguide/extending.html https://groups.google.com/d/msg/celery-users/3fs0ocREYqw/C7U1lCAp56sJ

closes #2519 https://pulp.plan.io/issues/2519 closes #2516 https://pulp.plan.io/issues/2516

Revision fd19f890 View on GitHub
Added by dalley almost 4 years ago

Workers write their own hearbeat records to database.

All workers will write their own records to the database instead of relying on pulp_celerybeat to do so for them using celery heartbeats.

This patch makes use of the Consumer blueprint that celery runs at the start time of a worker. An extra boot step has been added which sets a timer to periodically update the worker record in the database.

http://docs.celeryproject.org/en/master/userguide/extending.html https://groups.google.com/d/msg/celery-users/3fs0ocREYqw/C7U1lCAp56sJ

closes #2519 https://pulp.plan.io/issues/2519 closes #2516 https://pulp.plan.io/issues/2516

History

#1 Updated by bmbouter about 4 years ago

We generate our own timestamps and only use the worker name from the heartbeat. Instead of logging the heartbeat we receive, we should create our own heartbeat log line with the worker name and timestamp that we are using (the one we generate).

#2 Updated by bizhang about 4 years ago

  • Sprint/Milestone set to 31
  • Severity changed from 2. Medium to 1. Low
  • Triaged changed from No to Yes

#3 Updated by dalley about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#4 Updated by mhrivnak almost 4 years ago

  • Sprint/Milestone changed from 31 to 32

#5 Updated by dalley almost 4 years ago

  • Status changed from ASSIGNED to NEW

#6 Updated by dalley almost 4 years ago

  • Assignee deleted (dalley)

#7 Updated by dalley almost 4 years ago

Unassigning myself for grooming - jumped the gun a bit

#8 Updated by dalley almost 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

Assigning it to myself - will be resolved by the fix for Issue 2519 - https://github.com/pulp/pulp/pull/2922

No further effort needed

#10 Updated by dalley almost 4 years ago

  • Status changed from ASSIGNED to POST

#11 Updated by dalley almost 4 years ago

  • Status changed from POST to MODIFIED

#13 Updated by pthomas@redhat.com almost 4 years ago

Manual verification as per smash issue #513

#14 Updated by semyers almost 4 years ago

  • Platform Release set to 2.13.0

#15 Updated by pcreech over 3 years ago

  • Status changed from MODIFIED to 5

#16 Updated by pcreech over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#17 Updated by dalley over 3 years ago

  • Status changed from CLOSED - CURRENTRELEASE to CLOSED - WONTFIX

See discussion in the associated bugzilla. Revisited due to FailedQA, but we recognized that it is perfectly fine to remain the way it is currently, and isn't an issue.

#18 Updated by dalley over 3 years ago

  • Status changed from CLOSED - WONTFIX to CLOSED - NOTABUG

I suppose this should actually be "NOTABUG" instead of "WONTFIX"

#19 Updated by bmbouter almost 3 years ago

  • Sprint set to Sprint 16

#20 Updated by bmbouter almost 3 years ago

  • Sprint changed from Sprint 16 to Sprint 14

#21 Updated by bmbouter almost 3 years ago

  • Sprint/Milestone deleted (32)

#22 Updated by bmbouter almost 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF