Pulp

Pulp 3's pulp.tasking package does not work as-is with celery worker processes. Some re-arrangement has to happen to ensure that no pulp/django code gets used until the celery app has been instantiated and done it's "fixup" of django.

I would also add that because celery's django "fixup" does django.setup() we could not use it in our code. (Because the first and easy thing one may want to do is to put django.setup into our code.)

If you think it is an implementation detail, I am ok to skip it. The idea is just to introduce more reasoning here.

I think the current thinking is to move the task code itself moving to pulp.app.task. The task could be clarified a little bit on the new location. I imaging all tasks would go there including all three of these task types:

• tasks user know about (descendants of PulpUserFacingTask)
• tasks users care about directly but don't see (we may not have many of these in pulp3 but in 2.y these were things like reaper tasks. They are not descendants of PulpUserFacingTask)
• tasks and supporting classes that they don't need to know about (_queue_reserve_resource, PulpUserFacingTask, etc)

If all ^ move to pulp.app.task Wouldn't pulp.tasking only be left with the things that are part of pulp.tasking.services? My hope is that we can just have pulp.tasking and not pulp.tasking.<anotherlayerhere>.

My understanding of the "current thinking" is a little different, but not a lot. :)

The base pulp tasks, like PulpTask and UserFacingTask, in addition to the reservation-related tasks, are somewhat coupled to the "services" code that manages workers, and the celery app itself. They use the celery controller, create and delete working directories, etc. Likewise, the "services" code needs the ability to cancel tasks, so it uses the task cancellation code.

More generally, I think there is a basic set of tooling that is necessary to provide the tasking features we need. That tooling includes the base tasks, the reservation logic, and some worker management code. All of that is related, and to some extent inter-dependent. I think it makes sense for it to be in one distinct place. It's also worth noting that there's nothing pulp-specific about this functionality.

Then there are actual pulp tasks that get stuff done with pulp data. They only use the above functionality. I think these tasks are logically different from the tooling described above.

Thus the current in-flight layout is:

pulp.tasking contains the general tasking features we need, including the celery app itself
pulp.tasking.services has service-management code.
pulp.tasking.tasks has the base task classes, reservation tasks, cancellation, etc.

pulp.app.tasks has pulp tasks that do stuff with pulp data

I don't think there is a circular import concern as long as our model layer doesn't start importing tasks. That said, one interesting idea would be to make pulp.tasking a full django app and move the related models into it. I'm not sure what the implications would be for the REST API. But this would be the fullest extent of keeping the task system self-contained and separate from the tasks that use it. I'm not sold on it, but it's an idea.

To point out one quick variation on your last idea mhrivnak, pulp.tasking could become a Django app maybe named 'pulp_tasking' or something like that. pulp_app would require pulp_tasking to run and in all runtimes it would be present. This would allow those tasks to become @shared_tasks and we could then use the django+celery discovery to avoid importing pulp.tasking.tasks from pulp.app.tasks.

This is a subset of your idea since the proposal was to do all ^ but also move the models. Maybe they should move too (what you suggested).

mhrivnak: would you want to take the "in-flight" plan from comment 4, revise it based on our latest discussion and post it to the main task description?

I think this is ready to be groomed.