Pulp

I scheduled a puppet repo publish, and when it runs, I get two tasks in the task list:

Operations:
Resources:
State: Successful
Start Time: 2014-07-17T18:15:21Z
Finish Time: 2014-07-17T18:15:22Z
Task Id: 8bddd2ff-1edc-47e8-8cc2-45c2a6a59df4

Operations: publish
Resources: bug (repository)
State: Successful
Start Time: 2014-07-17T18:15:22Z
Finish Time: 2014-07-17T18:15:22Z
Task Id: 9495845a-a13d-41e4-87ce-c42f6504af88

I'm not sure why the first one is showing up. I think that probably some celery tasks in pulp.server.tasks.* were incorrectly given our Task base class, which tracks them in this way. Looking at the function pulp.server.tasks.repository.publish, it has the task decorator with "base=Task", which I think is incorrect. That would explain the behavior I'm seeing.

+ This comment was cloned from Bugzilla #1120671 comment 1 +

***** Bug 1131537 has been marked as a duplicate of this bug. ***

+ This comment was cloned from Bugzilla #1120671 comment 2 +

Looking at my own task list I determined that at least some of the tasks without resources and operation are the reaper and monthly tasks.

https://github.com/pulp/pulp/pull/1216

+ This comment was cloned from Bugzilla #1120671 comment 3 +

Merged into 2.5-dev

+ This comment was cloned from Bugzilla #1120671 comment 4 +

fixed in pulp 2.6.0-0.2.beta

+ This comment was cloned from Bugzilla #1120671 comment 5 +

Looking at the code change, I'm not convinced this fixed the problem as I was able to originally reproduce it. I don't see anything in this change that would affect a sync or publish.

Digging deeper, I think pulp.server.tasks.repository.* has been misunderstood. The tasks in here for "sync_with_auto_publish" and "publish" are not meant to be "public-facing" tasks that inherit from pulp's Task class. These are meant to be very simple functions that calculate tags and resources, then call the actual pulp task with that info. They are normal celery tasks so that the scheduler can queue them. When called from a web handler, these should be called synchronously.

See here for the original intent: https://github.com/pulp/pulp/commit/452e8249b1f89dad3170d9cffa9f0cfdbcb8f56e#diff-29

The current publish task looks correct, although it should not have the Task class passed to its decorator.

The current sync_with_auto_publish seems all wrong. It should call pulp.server.manager.repo.sync.sync() in a similar way that the publish function does. This task should only calculate tags and resources, then pass them to the sync task from the manager module. All other sync-related logic should be in the manager.

This is probably the same issue that was reported here, but we should confirm that before closing it as a dup: https://bugzilla.redhat.com/show_bug.cgi?id=1180677

+ This comment was cloned from Bugzilla #1120671 comment 6 +

Dennis, can you confirm Comment 6 and close https://bugzilla.redhat.com/show_bug.cgi?id=1180677 as a dup of this once you verify that this is the same issue? Thanks,

+ This comment was cloned from Bugzilla #1120671 comment 7 +

https://github.com/pulp/pulp/pull/1566

+ This comment was cloned from Bugzilla #1120671 comment 8 +

***** Bug 1180677 has been marked as a duplicate of this bug. ***

+ This comment was cloned from Bugzilla #1120671 comment 9 +

https://github.com/pulp/pulp/pull/1588

+ This comment was cloned from Bugzilla #1120671 comment 10 +

2.6.0-0.7.beta

+ This comment was cloned from Bugzilla #1120671 comment 11 +

rpm -qa pulp-server

pulp-server-2.6.0-0.7.beta.el6.noarch

pulp-admin tasks list

Operations: publish
Resources: gena (repository)
State: Successful
Start Time: 2015-02-13T22:37:23Z
Finish Time: 2015-02-13T22:37:23Z
Task Id: 48e5e874-6497-43a6-8be1-4c296327998a

Operations: publish
Resources: gena (repository)
State: Successful
Start Time: 2015-02-13T22:37:23Z
Finish Time: 2015-02-13T22:37:24Z
Task Id: 3425c0f2-c7c6-43f8-9a56-910b3702c7c8

pulp-admin rpm repo sync schedules create --repo-id lelik --schedule=2015-02-16T12:16:00Z/PT1M

Schedule successfully created

pulp-admin rpm repo sync schedules list --repo-id lelik

--------------------------------------------------------------------
Schedules
--------------------------------------------------------------------

Schedule: 2015-02-16T12:16:00Z/PT1M
Id: 54e22e18422f0716182e0395
Enabled: True
Next Run: 2015-02-16T17:51:30Z

date

Mon Feb 16 12:51:35 EST 2015

date -s "16 FEB 2015 17:51:00"

Mon Feb 16 17:51:00 EST 2015

pulp-admin rpm repo sync schedules list --repo-id lelik

--------------------------------------------------------------------
Schedules
--------------------------------------------------------------------

Schedule: 2015-02-16T12:16:00Z/PT1M
Id: 54e22e18422f0716182e0395
Enabled: True
Next Run: 2015-02-16T22:52:00Z

pulp-admin tasks list

Operations: publish
Resources: gena (repository)
State: Successful
Start Time: 2015-02-13T22:37:23Z
Finish Time: 2015-02-13T22:37:23Z
Task Id: 48e5e874-6497-43a6-8be1-4c296327998a

Operations: publish
Resources: gena (repository)
State: Successful
Start Time: 2015-02-13T22:37:23Z
Finish Time: 2015-02-13T22:37:24Z
Task Id: 3425c0f2-c7c6-43f8-9a56-910b3702c7c8

Operations: publish
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-16T22:53:26Z
Finish Time: 2015-02-16T22:53:52Z
Task Id: 41b7bbbd-1e75-4580-8945-324de4efe4b

pulp-admin rpm repo sync schedules list --repo-id lelik

--------------------------------------------------------------------
Schedules
--------------------------------------------------------------------

Schedule: 2015-02-16T12:16:00Z/PT1M
Id: 54e22e18422f0716182e0395
Enabled: True
Next Run: 2015-02-16T23:01:00Z

pulp-admin tasks list

Operations: publish
Resources: lelik (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 5559e1c9-3ea6-4a69-bc0a-5bc591efc63c

Operations: sync
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T00:09:13Z
Finish Time: 2015-02-17T00:12:29Z
Task Id: d2c1d5d9-24f9-4404-bcde-16c38bc942be

Operations: publish
Resources: lelik (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 4dd03f85-6540-4d5f-aa99-9d11eef4b631

Operations: sync
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T00:12:29Z
Finish Time: 2015-02-17T00:15:47Z
Task Id: 3c00b9ea-c0d8-4886-918c-91a1edad9681

Operations: publish
Resources: lelik (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: e6de03dd-d851-4978-9d24-3366e07b6e80

Operations: sync
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T00:15:47Z
Finish Time: 2015-02-17T00:19:08Z
Task Id: c64edc97-8642-4b0f-9129-ae46a00d2e16

Operations: sync
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T00:19:08Z
Finish Time: 2015-02-17T00:22:25Z
Task Id: f7d2ef1e-b404-4109-b43b-eb8a24389f1f

Operations: sync
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T00:22:25Z
Finish Time: 2015-02-17T00:25:42Z
Task Id: 0bde0758-c9de-4839-80da-a2ed24f21103

Operations: sync
Resources: lelik (repository)
State: Running
Start Time: 2015-02-17T00:25:42Z
Finish Time: Incomplete
Task Id: e8cec59b-1eb9-4b6c-b935-fcdc2db9ed6f

Operations: publish
Resources: lelik (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: a87ef383-8e89-4fcd-8e53-a1dc1a478615

pulp-admin rpm repo sync schedules list --repo-id lelik

--------------------------------------------------------------------
Schedules
--------------------------------------------------------------------

Schedule: 2015-02-16T12:16:00Z/PT1M
Id: 54e22e18422f0716182e0395
Enabled: True
Next Run: 2015-02-17T00:29:00Z

+ This comment was cloned from Bugzilla #1120671 comment 12 +

and for a regular sync task logging:

pulp-admin tasks list

...

Operations: publish
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T07:05:21Z
Finish Time: 2015-02-17T07:05:48Z
Task Id: 132f73ed-b838-4dc9-9031-8278756bd041

Operations: publish
Resources: lelik (repository)
State: Successful
Start Time: 2015-02-17T07:05:48Z
Finish Time: 2015-02-17T07:06:15Z
Task Id: 81b76bff-cd61-4377-b501-cdcedf0c3461

Operations: reaper
Resources:
State: Successful
Start Time: 2015-02-17T07:24:00Z
Finish Time: 2015-02-17T07:24:00Z
Task Id: e66eb25c-2ce2-47ec-83eb-f31a61cf5d0b

Operations: reaper
Resources:
State: Successful
Start Time: 2015-02-17T13:24:00Z
Finish Time: 2015-02-17T13:24:00Z
Task Id: 40fdcd19-e6f5-4f90-9c5e-cfde86e6c5d2

pulp-admin rpm repo list

--------------------------------------------------------------------
RPM Repositories
--------------------------------------------------------------------

Id: gena
Display Name: gena
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Rpm: 32

Id: bobik
Display Name: bobik
Description: None
Content Unit Counts:
Drpm: 3
Rpm: 37
Yum Repo Metadata File: 1

Id: lelik
Display Name: lelik
Description: None
Content Unit Counts:
Distribution: 1
Package Category: 14
Package Group: 212
Rpm: 6518

pulp-admin rpm repo sync run --repo-id gena

--------------------------------------------------------------------
Synchronizing Repository [gena]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[-]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

pulp-admin tasks list

...
Operations: reaper
Resources:
State: Successful
Start Time: 2015-02-17T07:24:00Z
Finish Time: 2015-02-17T07:24:00Z
Task Id: e66eb25c-2ce2-47ec-83eb-f31a61cf5d0b

Operations: reaper
Resources:
State: Successful
Start Time: 2015-02-17T13:24:00Z
Finish Time: 2015-02-17T13:24:00Z
Task Id: 40fdcd19-e6f5-4f90-9c5e-cfde86e6c5d2

Operations: sync
Resources: gena (repository)
State: Successful
Start Time: 2015-02-17T22:24:26Z
Finish Time: 2015-02-17T22:24:31Z
Task Id: 1fe2c77b-3616-43fa-a8d5-85e2c19aff32

Operations: publish
Resources: gena (repository)
State: Successful
Start Time: 2015-02-17T22:24:31Z
Finish Time: 2015-02-17T22:24:32Z
Task Id: e40e8f70-fc81-4949-bc80-a8a188b9fdf6

+ This comment was cloned from Bugzilla #1120671 comment 13 +