Issue #2930
pulp_celerybeat keeps going offline
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 23
Quarter:
Description
I see the following in my logs when I come back to my dev environment after a while.
[2017-07-23 16:39:06,658: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:39:11,607: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:46:35,493: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:35,496: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,547: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,547: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,577: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,577: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,603: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,603: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,635: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:46:36,636: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:46:41,608: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:49:23,592: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:23,592: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,675: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,675: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,686: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,687: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,699: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,699: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,724: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:49:24,724: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:49:29,701: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:52:01,564: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:01,564: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,606: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,607: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,628: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,629: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,642: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,642: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,669: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:52:02,669: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:52:07,634: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
History
#1
Updated by ttereshc over 3 years ago
- Sprint/Milestone set to 42
- Triaged changed from No to Yes
#2
Updated by fdobrovo over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to fdobrovo
#3
Updated by dkliban@redhat.com over 3 years ago
- Status changed from ASSIGNED to CLOSED - WORKSFORME
This issue has stopped presenting itself. It probably has to do with my laptop going into hibernation and the network going offline.
#4
Updated by fdobrovo over 3 years ago
- Status changed from CLOSED - WORKSFORME to CLOSED - WONTFIX
This happens whenever there is a relative forward time jump. This can happen if the virtual machine is temporally suspended due to host hibernation.
#5
Updated by bmbouter almost 3 years ago
- Sprint set to Sprint 23
#6
Updated by bmbouter almost 3 years ago
- Sprint/Milestone deleted (
42)
#7
Updated by daviddavis over 1 year ago
- Sprint/Milestone set to 3.0.0
#8
Updated by bmbouter over 1 year ago
- Tags deleted (
Pulp 3)
Please register to edit this issue