Actions
Issue #2930
closed
pulp_celerybeat keeps going offline
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 23
Quarter:
Description
I see the following in my logs when I come back to my dev environment after a while.
[2017-07-23 16:39:06,658: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:39:11,607: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:46:35,493: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:35,496: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,547: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,547: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,577: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,577: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,603: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,603: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,635: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:46:36,636: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:46:41,608: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:49:23,592: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:23,592: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,675: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,675: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,686: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,687: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,699: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,699: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,724: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:49:24,724: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:49:29,701: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:52:01,564: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:01,564: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,606: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,607: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,628: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,629: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,642: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,642: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,669: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:52:02,669: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:52:07,634: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
Updated by ttereshc over 7 years ago
- Sprint/Milestone set to 42
- Triaged changed from No to Yes
Updated by fdobrovo over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to fdobrovo
Updated by dkliban@redhat.com over 7 years ago
- Status changed from ASSIGNED to CLOSED - WORKSFORME
This issue has stopped presenting itself. It probably has to do with my laptop going into hibernation and the network going offline.
Updated by fdobrovo over 7 years ago
- Status changed from CLOSED - WORKSFORME to CLOSED - WONTFIX
This happens whenever there is a relative forward time jump. This can happen if the virtual machine is temporally suspended due to host hibernation.
Actions
.