Issue #2930: pulp_celerybeat keeps going offline - Pulp

Send by e-mail

Issue #2930

pulp_celerybeat keeps going offline

Added by dkliban@redhat.com about 2 years ago. Updated 6 months ago.

Status:

CLOSED - WONTFIX

Priority:

Normal

Assignee:

fdobrovo

Category:

Sprint/Milestone:

3.0

Start date:

Due date:

Severity:

2. Medium

Version:

Platform Release:

Blocks Release:

OS:

Backwards Incompatible:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

QA Contact:

Complexity:

Smash Test:

Verified:

Verification Required:

Sprint:

Sprint 23

Description

I see the following in my logs when I come back to my dev environment after a while.

[2017-07-23 16:39:06,658: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:39:11,607: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:46:35,493: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:35,496: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,547: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,547: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,577: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,577: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,603: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:46:36,603: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:46:36,635: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:46:36,636: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:46:41,608: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:49:23,592: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:23,592: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,675: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,675: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,686: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,687: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,699: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:49:24,699: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:49:24,724: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:49:24,724: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:49:29,701: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance
[2017-07-23 16:52:01,564: ERROR/MainProcess] Worker 'reserved_resource_worker_1@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:01,564: ERROR/MainProcess] The worker named reserved_resource_worker_1@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,606: ERROR/MainProcess] Worker 'reserved_resource_worker_2@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,607: ERROR/MainProcess] The worker named reserved_resource_worker_2@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,628: ERROR/MainProcess] Worker 'resource_manager@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,629: ERROR/MainProcess] The worker named resource_manager@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,642: ERROR/MainProcess] Worker 'celerybeat@dev.example.com' has gone missing, removing from list of workers
[2017-07-23 16:52:02,642: ERROR/MainProcess] The worker named celerybeat@dev.example.com is missing. Canceling the tasks in its queue.
[2017-07-23 16:52:02,669: ERROR/MainProcess] There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
[2017-07-23 16:52:02,669: ERROR/MainProcess] There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
[2017-07-23 16:52:07,634: WARNING/MainProcess] Failover occurred: 'celerybeat@dev.example.com' is now the primary celerybeat instance