Project

Profile

Help

Issue #3887

Restarting pulp_celerybeat weekly or so causes pulp.server.maintenance.monthly to not get scheduled

Added by ttereshc over 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.17.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Documentation, Pulp 2
Sprint:
Sprint 41
Quarter:

Description

Symptoms

No signs of `pulp.server.maintenance.monthly` in the logs for the last 30 days.
If task is triggered manually, it works properly and orphaned applicability profiles got cleaned up.

Impact

Having a lot of orphaned applicability profiles triggers a lot of no-op applicability tasks, it can slow down the whole system if there are too many of those tasks. E.g. A repo with many bound consumers (30K unique profiles) will trigger 3K tasks.

Root Cause

Pulp users Celery's periodic tasks feature for its dispatching. The monthly maintenance is dispatched every 30 days. Unfortunately Celery measues this as 30 days since the pulp_celerybeat process started. See the celery docs This means that if you restart your pulp_celerybeat every now and then, you'll end up with a lot of orphaned applicability profiles.

Resolution

The recommendation is to use cron to run a script monthly like:

from pulp.server.maintenance.monthly import queue_monthly_maintenance
queue_monthly_maintenance.apply_async()

Related issues

Related to Pulp - Issue #3908: Add pulp-maintenance package CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 97331ca0 View on GitHub
Added by bmbouter over 3 years ago

Add docs on maintenance jobs

This adds:

  • An extra configuration section
  • A troubleshooting section with symptom descriptions

https://pulp.plan.io/issues/3887 closes #3887

Revision 5af7d170 View on GitHub
Added by bmbouter over 3 years ago

Use the correct command for maintenance dispatch

https://pulp.plan.io/issues/3887 re #3887

History

#1 Updated by ttereshc over 3 years ago

  • Description updated (diff)

#2 Updated by ttereshc over 3 years ago

  • Description updated (diff)

#4 Updated by CodeHeeler over 3 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes
  • Sprint set to Sprint 40

#5 Updated by ttereshc over 3 years ago

Cron configuration to make this task run regularly is enough to fix this task.
The suggestion is to put it into /etc/cron.monthly and ship this config with rpm installation.

#6 Updated by bmbouter over 3 years ago

ttereshc excellent! Also remove this part of the code with that change too: https://github.com/pulp/pulp/blob/2-master/server/pulp/server/async/celery_instance.py#L30-L33

#7 Updated by bmbouter over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

#8 Updated by rchan over 3 years ago

  • Sprint changed from Sprint 40 to Sprint 41

#10 Updated by bmbouter over 3 years ago

This is working for me. I have the latest branch head of pulp/pulp:2-master and pulp/pulp_rpm:2-master. I applied the following diff to test it:

diff --git a/server/pulp/server/async/celery_instance.py b/server/pulp/server/async/celery_instance.py
index eab2ea3dd..9b486e8a5 100644
--- a/server/pulp/server/async/celery_instance.py
+++ b/server/pulp/server/async/celery_instance.py
@@ -29,7 +29,7 @@ CELERYBEAT_SCHEDULE = {
     },
     'monthly_maintenance': {
         'task': 'pulp.server.maintenance.monthly.queue_monthly_maintenance',
-        'schedule': timedelta(days=30),
+        'schedule': timedelta(seconds=60),
         'args': tuple(),
     },
     'download_deferred_content': {
diff --git a/server/pulp/server/maintenance/monthly.py b/server/pulp/server/maintenance/monthly.py
index 58e51f247..365496ea0 100644
--- a/server/pulp/server/maintenance/monthly.py
+++ b/server/pulp/server/maintenance/monthly.py
@@ -10,6 +10,7 @@ def queue_monthly_maintenance():
     """
     Create an itinerary for monthly task
     """
+    raise Exception('oh no!')
     tags = [action_tag('monthly')]
     monthly_maintenance.apply_async(tags=tags)

Then I got the following output upon startup:

kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
celery.beat:INFO: Scheduler: Sending due task monthly_maintenance (pulp.server.maintenance.monthly.queue_monthly_maintenance)
celery.worker.strategy:INFO: Received task: pulp.server.maintenance.monthly.queue_monthly_maintenance[e465c5d5-4385-4dcb-89ff-24013669c252]
celery.app.trace:ERROR: [e465c5d5] (18124-49472) Task pulp.server.maintenance.monthly.queue_monthly_maintenance[e465c5d5-4385-4dcb-89ff-24013669c252] raised unexpected: Exception('oh no!',)
celery.app.trace:ERROR: [e465c5d5] (18124-49472) Traceback (most recent call last):
celery.app.trace:ERROR: [e465c5d5] (18124-49472)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [e465c5d5] (18124-49472)     R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [e465c5d5] (18124-49472)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 107, in __call__
celery.app.trace:ERROR: [e465c5d5] (18124-49472)     return super(PulpTask, self).__call__(*args, **kwargs)
celery.app.trace:ERROR: [e465c5d5] (18124-49472)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
celery.app.trace:ERROR: [e465c5d5] (18124-49472)     return self.run(*args, **kwargs)
celery.app.trace:ERROR: [e465c5d5] (18124-49472)   File "/home/vagrant/devel/pulp/server/pulp/server/maintenance/monthly.py", line 13, in queue_monthly_maintenance
celery.app.trace:ERROR: [e465c5d5] (18124-49472)     raise Exception('oh no!')
celery.app.trace:ERROR: [e465c5d5] (18124-49472) Exception: oh no!

I am using Celery 4.0.2-3 with python2-celery-4.0.2-3.fc27.noarch.

#11 Updated by bmbouter over 3 years ago

  • Subject changed from Periodic task pulp.server.maintenance.monthly doesn't seem to get scheduled to Restarting pulp_celerybeat weekly or so causes pulp.server.maintenance.monthly to not get scheduled
  • Description updated (diff)
  • Tags Documentation added

Rewriting based on root cause discovery. This will become a docs bug explaining the recommended workaround.

#12 Updated by bmbouter over 3 years ago

  • Status changed from ASSIGNED to POST

#13 Updated by bmbouter over 3 years ago

  • Status changed from POST to MODIFIED

#14 Updated by bmbouter over 3 years ago

  • Related to Issue #3908: Add pulp-maintenance package added

#15 Updated by dkliban@redhat.com about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Platform Release set to 2.17.0

#16 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF