Project

Profile

Help

Issue #922

If a sync, publish, or consumer schedule is updated to change the time it runs, celerybeat does not run it at the new start time

Added by rbarlow over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.6.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

It seems I cannot update an existing sync, publish, or consumer action schedule. For example, I had an existing schedule, and I wanted to change it to run next in about four minutes (so that it's more than 90 seconds out, giving the scheduler enough time to notice the change). Here is me listing the schedule, and updating it to a new value:

[vagrant@cloud-basebox pulp]$ pulp-admin rpm repo sync schedules list --repo-id zoo
+----------------------------------------------------------------------+
                               Schedules
+----------------------------------------------------------------------+

Schedule: 2015-04-27T21:24:00Z/P1D
Id:       553be0de7517d40591882021
Enabled:  True
Next Run: 2015-04-28T18:48:00Z

[vagrant@cloud-basebox pulp]$ pulp-admin rpm repo sync schedules update --repo-id zoo --schedule-id 553be0de7517d40591882021 --schedule 2015-04-27T21:32:00Z/P1D
Successfully updated schedule

[vagrant@cloud-basebox pulp]$ date -u
Mon Apr 27 21:28:44 UTC 2015

The Next Run continues to show 2015-04-28T18:48:00Z well past the new schedule start time, and according to the logs the new schedule never fired (I didn't see a sync happen in the log.)

A workaround is to delete the existing schedule and create a new schedule with the desired start time.

Associated revisions

Revision 472b84cb View on GitHub
Added by bmbouter over 6 years ago

Updates to schedules are now respected by celerybeat

The schedule field was being updated, but next_run and first_run were not being updated. These values are used to determine if a schedued task should be run so updating them is necessary.

fixes #922 https://pulp.plan.io/issues/922

Revision 472b84cb View on GitHub
Added by bmbouter over 6 years ago

Updates to schedules are now respected by celerybeat

The schedule field was being updated, but next_run and first_run were not being updated. These values are used to determine if a schedued task should be run so updating them is necessary.

fixes #922 https://pulp.plan.io/issues/922

Revision 7fd27725 View on GitHub
Added by ipanova@redhat.com about 6 years ago

Adding saslwrapper to the external deps list.

closes #922 https://pulp.plan.io/issues/992

Revision 7fd27725 View on GitHub
Added by ipanova@redhat.com about 6 years ago

Adding saslwrapper to the external deps list.

closes #922 https://pulp.plan.io/issues/992

History

#1 Updated by rbarlow over 6 years ago

  • Description updated (diff)

Here is some more information I gathered about this. I updated the schedule again to be a few minutes in the future:

$ pulp-admin rpm repo sync schedules update --repo-id zoo --schedule-id 553be0de7517d40591882021 --schedule 2015-04-27T22:20:00Z/P1D 
Successfully updated schedule

Then I configured celerybeat to run in Debug logging, and saw this:

Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: root:DEBUG: one or more enabled schedules has been updated
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: pulp.server.async.scheduler:DEBUG: loading schedules from app
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: pulp.server.async.scheduler:DEBUG: loading schedules from DB
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: pulp.server.async.scheduler:DEBUG: loaded 2 schedules
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: pulp.server.db.model.dispatch:DEBUG: not running task 553eaee47517d419c494b574: 84931 seconds remaining
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: pulp.server.db.model.dispatch:DEBUG: not running task 553be0de7517d40591882021: 73831 seconds remaining
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: pulp.server.async.worker_watcher:DEBUG: received 'scheduler-event' from scheduler@cloud-basebox.localdomain at time: 2015-04-27 22:17:28.973679
Apr 27 22:17:28 cloud-basebox.localdomain pulp[6978]: celery.beat:DEBUG: beat: Waking up in 1.50 minutes.

Note that the scheduler saw the change at 22:17, which was before the next run at 22:20. However, note that it still thinks the next run for that schedule is 73,831 seconds away rather than ~2.5 minutes.

#2 Updated by bmbouter over 6 years ago

Is it possible this is related to #516?

#3 Updated by rbarlow over 6 years ago

On 04/28/2015 10:48 AM, bmbouter wrote:

Is it possible this is related to #516 <https://pulp.plan.io/issues/516>?

It is possible, though I'm not certain at this point. I didn't try to
adjust the frequency, but instead I tried to update the start time. That
bug seems to be about adjusting the frequency, so it may or may not be
the same. It's worth looking into, thanks for the pointer!

--
Randy Barlow

#4 Updated by mhrivnak over 6 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Version set to Master
  • Triaged changed from No to Yes

Investigate whether this affects 2.6.

#5 Updated by bmbouter over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

#7 Updated by bmbouter over 6 years ago

  • Subject changed from If a sync schedule is updated to change the time it runs, the scheduler does not run it at the new start time to If a sync, publish, or consumer schedules are updated to change the time it runs, the celerybeat does not run it at the new start time
  • Description updated (diff)

QE to verify this, please do the following.

0) Install a fully working version of Pulp and start it up
1) Create a zoo repo
2) Create a sync schedule for zoo that will sync very far in the future (ie: a date in 2016)
3) List that sync and verify it got created
4) Update the sync with pulp-admin to have it run a few minutes in the future (at least 120 seconds from the time you issue the command)
5) Verify that the sync runs at the new time you specified
6) Do the same steps 2-5 for a publish schedule and verify that
7) Install a consumer
8) Do steps 2-5 only with a consumer action and verify that type of schedule also updates.

#8 Updated by bmbouter over 6 years ago

  • Status changed from ASSIGNED to POST

#9 Updated by bmbouter over 6 years ago

  • Subject changed from If a sync, publish, or consumer schedules are updated to change the time it runs, the celerybeat does not run it at the new start time to If a sync, publish, or consumer schedule is updated to change the time it runs, celerybeat does not run it at the new start time

#10 Updated by bmbouter over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#11 Updated by bmbouter over 6 years ago

  • Platform Release set to 2.6.2

#12 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5

#14 Updated by pthomas@redhat.com over 6 years ago

  • Status changed from 5 to 6

verified
[root@mgmt12 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@mgmt12 ~]#
1. verified sync update works as expected
2. also verified consumer install schedule work

[root@mgmt12 ~]# pulp-admin rpm consumer  package  install schedules create --consumer-id patch --schedule 2016-04-27T22:20:00Z/P1D -n dog
Schedule successfully created

[root@mgmt12 ~]# 
[root@mgmt12 ~]# 
[root@mgmt12 ~]# pulp-admin rpm consumer  package  install schedules list --consumer-id patch 
+----------------------------------------------------------------------+
                               Schedules
+----------------------------------------------------------------------+

Schedule: 2015-05-20T15:05:00Z/P1D
Id:       555c9d62e8ec9860dcbdcadb
Enabled:  True
Next Run: 2015-05-21T15:05:00Z

Schedule: 2016-04-27T22:20:00Z/P1D
Id:       555ca3b3e8ec9860dbfe47fa
Enabled:  True
Next Run: 2016-04-27T22:20:00Z

[root@mgmt12 ~]# pulp-admin rpm consumer  package  install schedules update --consumer-id patch  --schedule-id 555ca3b3e8ec9860dbfe47fa  -s 2015-05-20T15:15:00Z/P1D
Successfully updated schedule

[root@mgmt12 ~]# pulp-admin rpm consumer  package  install schedules list --consumer-id patch 
+----------------------------------------------------------------------+
                               Schedules
+----------------------------------------------------------------------+

Schedule: 2015-05-20T15:05:00Z/P1D
Id:       555c9d62e8ec9860dcbdcadb
Enabled:  True
Next Run: 2015-05-21T15:05:00Z

Schedule: 2015-05-20T15:15:00Z/P1D
Id:       555ca3b3e8ec9860dbfe47fa
Enabled:  True
Next Run: 2015-05-20T15:15:00Z

<\pre>

#15 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#18 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF