Project

Profile

Help

Task #3522

Updated by milan over 6 years ago

h2. End to end design. 

 When the design reaches consensus, sub-tasks for core and plugins should be created. 

 h3. Component 1. Make Tasks Master/Detail. 

 * Change inheritance of Tasks 
 * Update TaskSerializer for master/detail 
 * override TaskSerializer.create to deploy tasks 
 * update apply_async_with_reservation to accept Tasks (the django model kind, "task_status" in this code)  
 * Update OperationPostponed to work with Core Tasks 

 h3. Component 2. Tasks relating to Update/Delete of other objects 
 Some objects cause the deployment of tasks for Update and Delete. (Importer, Publisher, Repository). From the user's perspective, this will be almost unchanged, except the href for the tasks will change: 

 Old: (Updates and Deletes) v3/tasks/1234 
 New: v3/tasks/core/updates/1234 
          v3/tasks/core/deletes/1234 

 This change will be in pulpcore, and will not affect plugins: 

 * create detail task for Update 
 * create detail task for Delete 
 * update viewset mixin for AsyncUpdate 
 * update viewset mixin for AsyncDelete 

 Allow the direct creation of Master/Detail Tasks that are connected to the celery tasking system, replacing the previous pattern of "action endpoints". 

 Old: POST /v3/importers/file/1234/sync/ repository=repohref 
 New POST v3/tasks/file/syncs/ importer=importerhref repository=repositoryhref 

 h3. Component 3: Plugins 

 * Create models for Detail Tasks 
 * Create serializers for Detail Tasks 
 ** 3 new class attrs: 
 *** reservation_structure 
 *** task_kwarg_structure 
 *** celery_task  
 ** new fields corresponding to arguments for celery task 
 * Create viewsets for Detail Tasks 
 * remove "action" endpoints 

 h2. Motivations 

 Some discussion on this: https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html 

 * Plugin code is simplified. (Plugin writers no longer need to deploy tasks. Detail Task classes    require new attributes (which are easily explained) but do not need to interact with the tasking system.) 
 * Task parameters are specified by the detail Task model, so validation is handled by the Detail Task serializer (href validation is free!) 
 * Task history becomes more valuable.  
 ** filter tasks based on task parameters like repository or importer 
 ** filter tasks based on created_resource, and see which importer was used 
 ** Easy to repeat tasks? 
 * Tasks will be viewable by plugin, and task type. 
 * Replace "action endpoints" with standard DRF object CRUD 

 Summary: The plugin writers will write less complex code, which is replaced by normal DRF objects (which are very similar to all the other work of writing plugins). 


 This has a few advantages which are summarized under the motivation section. 

 # Task parameters are specified by the Detail Task model. 

 ### Tasks are typed, so you can see whether a version was created with a SyncTask or an AddRemoveTask 

 
 h2. User Facing Comparison 

 h3. Object CRUD 

 Creation of Objects will be exactly the same. 

 Update/Delete of objects that require reservations will be identical except that the task href that is returned will be namespaced. 

 One possible downside of this pattern is that "actions" are deployed to Pulp by creating Tasks, but Update/Delete tasks are side effects. Though this could be confusing at first, I think it makes sense. When deploying an "action", the task creation is primary and the created_resources is secondary. When deploying and "update" or "delete", the object being affected is primary, and the task to do it is secondary.  

 <pre> 
 $ http PATCH http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/ name=r2 
 HTTP/1.0 202 Accepted 
 Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS 
 Content-Length: 148 
 Content-Type: application/json 
 Date: Fri, 23 Mar 2018 16:41:02 GMT 
 Server: WSGIServer/0.2 CPython/3.6.4 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 [ 
     { 
         "_href": "http://pulp3.dev:8000/api/v3/tasks/core/updates/824f60ff-13a1-4de6-91a7-a3f6dc23707b/", 
         "task_id": "824f60ff-13a1-4de6-91a7-a3f6dc23707b" 
     } 
 ] 

 $ http http://pulp3.dev:8000/api/v3/tasks/core/updates/824f60ff-13a1-4de6-91a7-a3f6dc23707b/ 
 HTTP/1.0 200 OK 
 Allow: GET, HEAD, OPTIONS 
 Content-Length: 348 
 Content-Type: application/json 
 Date: Fri, 23 Mar 2018 16:41:08 GMT 
 Server: WSGIServer/0.2 CPython/3.6.4 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 { 
     "_href": "http://pulp3.dev:8000/api/v3/tasks/core/updates/824f60ff-13a1-4de6-91a7-a3f6dc23707b/", 
     "created_resources": [], 
     "error": null, 
     "finished_at": "2018-03-23T16:41:03.452739Z", 
     "non_fatal_errors": [], 
     "started_at": "2018-03-23T16:41:03.408065Z", 
     "state": "completed", 
     "worker": "http://pulp3.dev:8000/api/v3/workers/fd319bbb-789e-462b-b87a-83143543330a/" 
 } 
 </pre> 

 h2. REST API overview 

 This change would enocurage consistency between plugins by making an "easy obvious way" to deploy and view tasks. This would encourage (but not require) the following urls 

 v3/tasks/docker/add-removes/ 
 v3/tasks/docker/syncs/ 
 v3/tasks/docker/publishes/ 
 v3/tasks/file/add-removes/ 
 v3/tasks/file/syncs/ 
 v3/tasks/file/publishes/ 
 v3/tasks/core/updates/ 
 v3/tasks/core/deletes/ 

 It is trivial to create "tiered viewsets". 

 v3/tasks/file/    <---------    list all file tasks (add-remove, syncs, publishes) 
 v3/tasks/        <-----------    list all tasks, which are .cast() and serialized 

 This will be much more obvious than how we are doing it now, because tasks are spread out. Sync has a place now, because a sync task is associated with an importer (v3/importers/file/1234/sync/). But what about plugins that want to allow multiple importers in a single sync? And what about tasks that don't use a plugin-defined object (like an importer)?  

 For example, rich dependencies don't need an importer, so the plugin writer is on their own to define everything, including dispatching their celery tasks and choosing and endpoint. 


 h3. Sync Task example 

 Deploy a sync task 
 <pre> 
  http http://pulp3.dev:8000/api/v3/tasks/file/syncs/ importer=http://pulp3.dev:8000/api/v3/importers/file/7a8866a4-f6f4-4ab3-ade6-678e2328b238/ repository=http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/ 
 HTTP/1.0 201 Created 
 Allow: GET, POST, HEAD, OPTIONS 
 Content-Length: 412 
 Content-Type: application/json 
 Date: Fri, 23 Mar 2018 16:47:17 GMT 
 Location: http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af-80e6-4b95-9a8c-f315eb6872cf/ 
 Server: WSGIServer/0.2 CPython/3.6.4 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 { 
     "_href": "http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af-80e6-4b95-9a8c-f315eb6872cf/", 
     "created_resources": [], 
     "error": null, 
     "finished_at": null, 
     "importer": "http://pulp3.dev:8000/api/v3/importers/file/7a8866a4-f6f4-4ab3-ade6-678e2328b238/", 
     "non_fatal_errors": [], 
     "repository": "http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/", 
     "started_at": null, 
     "state": "waiting", 
     "worker": null 
 } 
 </pre> 
 Retrieve Sync Task: 
 <pre> 
 ~ ❯ http http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af-80e6-4b95-9a8c-f315eb6872cf/ 
 HTTP/1.0 200 OK 
 Allow: GET, HEAD, OPTIONS 
 Content-Length: 624 
 Content-Type: application/json 
 Date: Fri, 23 Mar 2018 16:47:54 GMT 
 Server: WSGIServer/0.2 CPython/3.6.4 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 { 
     "_href": "http://pulp3.dev:8000/api/v3/tasks/file/syncs/9d78f7af80e64b959a8cf315eb6872cf/", 
     "created_resources": [ 
         "http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/versions/6/" 
     ], 
     "error": null, 
     "finished_at": "2018-03-23T16:47:18.484046Z", 
     "importer": "http://pulp3.dev:8000/api/v3/importers/file/7a8866a4-f6f4-4ab3-ade6-678e2328b238/", 
     "non_fatal_errors": [], 
     "repository": "http://pulp3.dev:8000/api/v3/repositories/7e91a5c3-4a96-4d27-b5f2-99ccc563eaab/", 
     "started_at": "2018-03-23T16:47:17.939148Z", 
     "state": "completed", 
     "worker": "http://pulp3.dev:8000/api/v3/workers/fd319bbb-789e-462b-b87a-83143543330a/" 
 } 

 </pre> 


 To make sure this was feasable, I've created proof of concept PRs for pulpcore and pulp_file. All of the proposed changes are proofed, but not all work is done. For example, sync is implemented, but not publish. 
 https://github.com/pulp/pulp/pull/3394 
 https://github.com/pulp/pulp_file/pull/61 

Back