Task #3522
Updated by milan about 6 years ago
h2. End to end design.
When the design reaches consensus, sub-tasks for core and plugins should be created.
h3. Component 1. Make Tasks Master/Detail.
* Change inheritance of Tasks
* Update TaskSerializer for master/detail
* override TaskSerializer.create to deploy tasks
* update apply_async_with_reservation to accept Tasks (the django model kind, "task_status" in this code)
* Update OperationPostponed to work with Core Tasks
h3. Component 2. Tasks relating to Update/Delete of other objects
Some objects cause the deployment of tasks for Update and Delete. (Importer, Publisher, Repository). From the user's perspective, this will be almost unchanged, except the href for the tasks will change:
Old: (Updates and Deletes) v3/tasks/1234
New: v3/tasks/core/updates/1234
v3/tasks/core/deletes/1234
This change will be in pulpcore, and will not affect plugins:
* create detail task for Update
* create detail task for Delete
* update viewset mixin for AsyncUpdate
* update viewset mixin for AsyncDelete
Allow the direct creation of Master/Detail Tasks that are connected to the celery tasking system, replacing the previous pattern of "action endpoints".
Old: POST /v3/importers/file/1234/sync/ repository=repohref
New POST v3/tasks/file/syncs/ importer=importerhref repository=repositoryhref
h3. Component 3: Plugins
* Create models for Detail Tasks
* Create serializers for Detail Tasks
** 3 new class attrs:
*** reservation_structure
*** task_kwarg_structure
*** celery_task
** new fields corresponding to arguments for celery task
* Create viewsets for Detail Tasks
* remove "action" endpoints
h2. Motivations
Some discussion on this: https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
* Plugin code is simplified. (Plugin writers no longer need to deploy tasks. Detail Task classes require new attributes (which are easily explained) but do not need to interact with the tasking system.)
* Task parameters are specified by the detail Task model, so validation is handled by the Detail Task serializer (href validation is free!)
* Task history becomes more valuable.
** filter tasks based on task parameters like repository or importer
** filter tasks based on created_resource, and see which importer was used
** Easy to repeat tasks?
* Tasks will be viewable by plugin, and task type.
* Replace "action endpoints" with standard DRF object CRUD
Summary: The plugin writers will write less complex code, which is replaced by normal DRF objects (which are very similar to all the other work of writing plugins).
This has a few advantages which are summarized under the motivation section.
# Task parameters are specified by the Detail Task model.
### Tasks are typed, so you can see whether a version was created with a SyncTask or an AddRemoveTask
h2. User Facing Comparison
h3. Object CRUD
Creation of Objects will be exactly the same.
Update/Delete of objects that require reservations will be identical except that the task href that is returned will be namespaced.
One possible downside of this pattern is that "actions" are deployed to Pulp by creating Tasks, but Update/Delete tasks are side effects. Though this could be confusing at first, I think it makes sense. When deploying an "action", the task creation is primary and the created_resources is secondary. When deploying and "update" or "delete", the object being affected is primary, and the task to do it is secondary.
<pre>
$ http PATCH http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/ name=r2
HTTP/1.0 202 Accepted
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Content-Length: 148
Content-Type: application/json
Date: Fri, 23 Mar 2018 16:41:02 GMT
Server: WSGIServer/0.2 CPython/3.6.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
[
{
"_href": "http://pulp3.dev:8000/api/v3/tasks/core/updates/824f60ff-13a1-4de6-91a7-a3f6dc23707b/",
"task_id": "824f60ff-13a1-4de6-91a7-a3f6dc23707b"
}
]
$ http http://pulp3.dev:8000/api/v3/tasks/core/updates/824f60ff-13a1-4de6-91a7-a3f6dc23707b/
HTTP/1.0 200 OK
Allow: GET, HEAD, OPTIONS
Content-Length: 348
Content-Type: application/json
Date: Fri, 23 Mar 2018 16:41:08 GMT
Server: WSGIServer/0.2 CPython/3.6.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_href": "http://pulp3.dev:8000/api/v3/tasks/core/updates/824f60ff-13a1-4de6-91a7-a3f6dc23707b/",
"created_resources": [],
"error": null,
"finished_at": "2018-03-23T16:41:03.452739Z",
"non_fatal_errors": [],
"started_at": "2018-03-23T16:41:03.408065Z",
"state": "completed",
"worker": "http://pulp3.dev:8000/api/v3/workers/fd319bbb-789e-462b-b87a-83143543330a/"
}
</pre>
h2. REST API overview
This change would enocurage consistency between plugins by making an "easy obvious way" to deploy and view tasks. This would encourage (but not require) the following urls
v3/tasks/docker/add-removes/
v3/tasks/docker/syncs/
v3/tasks/docker/publishes/
v3/tasks/file/add-removes/
v3/tasks/file/syncs/
v3/tasks/file/publishes/
v3/tasks/core/updates/
v3/tasks/core/deletes/
It is trivial to create "tiered viewsets".
v3/tasks/file/ <--------- list all file tasks (add-remove, syncs, publishes)
v3/tasks/ <----------- list all tasks, which are .cast() and serialized
This will be much more obvious than how we are doing it now, because tasks are spread out. Sync has a place now, because a sync task is associated with an importer (v3/importers/file/1234/sync/). But what about plugins that want to allow multiple importers in a single sync? And what about tasks that don't use a plugin-defined object (like an importer)?
For example, rich dependencies don't need an importer, so the plugin writer is on their own to define everything, including dispatching their celery tasks and choosing and endpoint.
h3. Sync Task example
Deploy a sync task
<pre>
http http://pulp3.dev:8000/api/v3/tasks/file/syncs/ importer=http://pulp3.dev:8000/api/v3/importers/file/7a8866a4-f6f4-4ab3-ade6-678e2328b238/ repository=http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/
HTTP/1.0 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 412
Content-Type: application/json
Date: Fri, 23 Mar 2018 16:47:17 GMT
Location: http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af-80e6-4b95-9a8c-f315eb6872cf/
Server: WSGIServer/0.2 CPython/3.6.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_href": "http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af-80e6-4b95-9a8c-f315eb6872cf/",
"created_resources": [],
"error": null,
"finished_at": null,
"importer": "http://pulp3.dev:8000/api/v3/importers/file/7a8866a4-f6f4-4ab3-ade6-678e2328b238/",
"non_fatal_errors": [],
"repository": "http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/",
"started_at": null,
"state": "waiting",
"worker": null
}
</pre>
Retrieve Sync Task:
<pre>
~ ❯ http http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af-80e6-4b95-9a8c-f315eb6872cf/
HTTP/1.0 200 OK
Allow: GET, HEAD, OPTIONS
Content-Length: 624
Content-Type: application/json
Date: Fri, 23 Mar 2018 16:47:54 GMT
Server: WSGIServer/0.2 CPython/3.6.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_href": "http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af80e64b959a8cf315eb6872cf/",
"created_resources": [
"http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/versions/6/"
],
"error": null,
"finished_at": "2018-03-23T16:47:18.484046Z",
"importer": "http://pulp3.dev:8000/api/v3/importers/file/7a8866a4-f6f4-4ab3-ade6-678e2328b238/",
"non_fatal_errors": [],
"repository": "http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/",
"started_at": "2018-03-23T16:47:17.939148Z",
"state": "completed",
"worker": "http://pulp3.dev:8000/api/v3/workers/fd319bbb-789e-462b-b87a-83143543330a/"
}
</pre>
To make sure this was feasable, I've created proof of concept PRs for pulpcore and pulp_file. All of the proposed changes are proofed, but not all work is done. For example, sync is implemented, but not publish.
https://github.com/pulp/pulp/pull/3394
https://github.com/pulp/pulp_file/pull/61
.