Project

Profile

Help

Actions
Send by e-mail Copy link

Task #2380

closed

Create a redmine task for each 2.y celery task to be converted to 3.0

Added by bmbouter about 8 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
ttereshc
Category:
-
Sprint/Milestone:
3.0.0
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 10
Quarter:

Description

We have roughly 20-30 celery tasks in Pulp 2.y. We need to have a redmine task created for each one tracking the conversion of that celery task to Pulp3.

To identify the list of celery tasks in 2.y I recommend inspecting the celery registry of a 2.y dev install. You can inspect the registry using a technique similar to what this[0] unit test does.

Each task needs to be moved out of the pulp.server and into somewhere. We should discuss on the issues themselves where each one should go. Also on Pulp3 all of the tasks are expected to be imported into pulp.tasking.registry which is where celery will discover them. Note that it's probably not a good idea to have the task code itself live in pulp.tasking.registry so where to house that code is something we should discuss.

The details of what the Pulp3 behaviors should be for that task can be discussed on each issue created. There are a few that will be deleted. Let's make issues for all the 2.y tasks and then we can identify ones to close on the individual issues.

[0]: https://github.com/bmbouter/pulp/blob/1fe832ce1d320129c680755f38cafae26f5159d0/server/test/unit/server/async/test_tasks.py#L1053-L1053

Related issues

Related to Pulp - Refactor #2154: Convert task system to use django modelsCLOSED - CURRENTRELEASEbmbouter

Actions
Related to Pulp - Task #2386: Convert celery task repository.download_deferred to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2388: Convert celery task repository.download_repo to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2389: Convert celery task applicability.batch_regenerate_applicability to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2390: Convert celery task applicability.regenerate_applicability_for_consumers to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2391: Convert celery task applicability.regenerate_applicability_for_repos to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2392: Convert celery task db.reaper.reap_expired_documents to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2393: Convert celery task monthly_maintenance to Pulp 3CLOSED - NOTABUG

Actions
Related to Pulp - Task #2394: Convert celery task content.orphan.delete_all_orphans to Pulp 3CLOSED - CURRENTRELEASEttereshc

Actions
Related to Pulp - Task #2395: Convert celery task content.orphan.delete_orphans_by_id to Pulp 3CLOSED - COMPLETE

Actions
Related to Pulp - Story #2396: As a user, I can remove orphaned content and related artifacts by content typeCLOSED - DUPLICATE

Actions
Related to Pulp - Task #2397: Convert celery task content.upload.import_uploaded_unit to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2398: Convert celery task repo.publish to Pulp 3CLOSED - CURRENTRELEASEdkliban@redhat.com

Actions
Related to Pulp - Task #2399: Convert celery task repo.sync to Pulp 3CLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Related to Pulp - Task #2400: Convert celery task repo.importer.remove_importer to Pulp 3CLOSED - CURRENTRELEASEmhrivnak

Actions
Related to Pulp - Task #2401: Convert celery task repo.importer.set_importer to Pulp 3CLOSED - DUPLICATE

Actions
Related to Pulp - Story #2402: As an API user, a call to update an Importer generates a TaskCLOSED - CURRENTRELEASEdkliban@redhat.com

Actions
Related to Pulp - Task #2403: Convert celery task repo.unit_association.associate_from_repo to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2404: Convert celery task repo.unit_association.unassociate_by_criteria to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2405: Convert celery task content.refresh_content_source to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2406: Convert celery task content.refresh_content_sourceS to Pulp 3CLOSED - WONTFIX

Actions
Related to Pulp - Task #2407: Convert celery task repository.delete to Pulp 3CLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Related to Pulp - Task #2408: Convert celery task repository.distributor_delete to Pulp 3CLOSED - CURRENTRELEASEttereshc

Actions
Related to Pulp - Story #2409: As an API user, a call to update an Publisher generates a TaskCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Actions
Copy link
 #1

Updated by mhrivnak about 8 years ago 

$ sudo -u apache celery inspect registered -A pulp.server.async.app
-> reserved_resource_worker-0@dev: OK
    * pulp.server.async.tasks.PulpTask
    * pulp.server.async.tasks.Task
    * pulp.server.async.tasks._queue_reserved_task
    * pulp.server.async.tasks._release_resource
    * pulp.server.controllers.repository.download_deferred
    * pulp.server.controllers.repository.download_repo
    * pulp.server.controllers.repository.queue_download_deferred
    * pulp.server.controllers.repository.queue_publish
    * pulp.server.db.reaper.queue_reap_expired_documents
    * pulp.server.db.reaper.reap_expired_documents
    * pulp.server.maintenance.monthly.monthly_maintenance
    * pulp.server.maintenance.monthly.queue_monthly_maintenance
    * pulp.server.managers.auth.permission.cud.grant
    * pulp.server.managers.auth.permission.cud.revoke
    * pulp.server.managers.auth.role.cud.add_permissions_to_role
    * pulp.server.managers.auth.role.cud.add_user_to_role
    * pulp.server.managers.auth.role.cud.create_role
    * pulp.server.managers.auth.role.cud.delete_role
    * pulp.server.managers.auth.role.cud.remove_permissions_from_role
    * pulp.server.managers.auth.role.cud.remove_user_from_role
    * pulp.server.managers.auth.role.cud.update_role
    * pulp.server.managers.consumer.agent.delete_queue
    * pulp.server.managers.consumer.applicability.batch_regenerate_applicability
    * pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers
    * pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos
    * pulp.server.managers.consumer.bind.bind
    * pulp.server.managers.consumer.bind.delete
    * pulp.server.managers.consumer.bind.unbind
    * pulp.server.managers.consumer.cud.register
    * pulp.server.managers.consumer.cud.unregister
    * pulp.server.managers.consumer.cud.update
    * pulp.server.managers.consumer.group.cud.associate
    * pulp.server.managers.consumer.group.cud.bind
    * pulp.server.managers.consumer.group.cud.create_consumer_group
    * pulp.server.managers.consumer.group.cud.delete_consumer_group
    * pulp.server.managers.consumer.group.cud.unassociate
    * pulp.server.managers.consumer.group.cud.unbind
    * pulp.server.managers.consumer.group.cud.update_consumer_group
    * pulp.server.managers.consumer.profile.create
    * pulp.server.managers.consumer.profile.delete
    * pulp.server.managers.consumer.profile.update
    * pulp.server.managers.content.orphan.delete_all_orphans
    * pulp.server.managers.content.orphan.delete_orphans_by_id
    * pulp.server.managers.content.orphan.delete_orphans_by_type
    * pulp.server.managers.content.upload.import_uploaded_unit
    * pulp.server.managers.repo.group.cud.associate
    * pulp.server.managers.repo.group.cud.create_and_configure_repo_group
    * pulp.server.managers.repo.group.cud.delete_repo_group
    * pulp.server.managers.repo.group.cud.unassociate
    * pulp.server.managers.repo.group.cud.update_repo_group
    * pulp.server.managers.repo.group.distributor.add_distributor
    * pulp.server.managers.repo.group.distributor.remove_distributor
    * pulp.server.managers.repo.group.distributor.update_distributor_config
    * pulp.server.managers.repo.group.publish.publish
    * pulp.server.managers.repo.importer.remove_importer
    * pulp.server.managers.repo.importer.set_importer
    * pulp.server.managers.repo.importer.update_importer_config
    * pulp.server.managers.repo.publish.publish
    * pulp.server.managers.repo.sync.sync
    * pulp.server.managers.repo.unit_association.associate_from_repo
    * pulp.server.managers.repo.unit_association.unassociate_by_criteria
    * pulp.server.tasks.consumer.install_content
    * pulp.server.tasks.consumer.uninstall_content
    * pulp.server.tasks.consumer.update_content
    * pulp.server.tasks.content.refresh_content_source
    * pulp.server.tasks.content.refresh_content_sources
    * pulp.server.tasks.repository.delete
    * pulp.server.tasks.repository.distributor_delete
    * pulp.server.tasks.repository.distributor_update
    * pulp.server.tasks.repository.sync_with_auto_publish
Actions
Copy link
 #2

Updated by mhrivnak about 8 years ago

I think these can be ignored, and not ported to pulp 3:

  1. auth tasks
  2. consumer tasks that aren't related to applicability
  3. repo group tasks
  4. Any task that was only used by the scheduler to queue other user-facing tasks. Example: queue_publish (also, I'm not sure why "queue_sync_with_auto_publish" is missing from this list)
Actions
Copy link
 #3

Updated by bmbouter about 8 years ago

Thanks mhrivnak. Yes all of those things should not be ported.

Also, these 4 are already ported so they can be ignored.

    * pulp.server.async.tasks.PulpTask
    * pulp.server.async.tasks.Task
    * pulp.server.async.tasks._queue_reserved_task
    * pulp.server.async.tasks._release_resource
Actions
Copy link
 #4

Updated by ttereshc about 8 years ago

mhrivnak wrote:

  1. Any task that was only used by the scheduler to queue other user-facing tasks. Example: queue_publish (also, I'm not sure why "queue_sync_with_auto_publish" is missing from this list)

It is there, the name is without "queue_" for some reason, just sync_with_auto_publish

Actions
Copy link
 #5

Updated by ttereshc about 8 years ago

This comment is just to track what celery tasks have been covered so far. Feel free to edit it accordingly.

  • task #2154 pulp.server.async.tasks.PulpTask
  • task #2154 pulp.server.async.tasks.Task
  • task #2154 pulp.server.async.tasks._queue_reserved_task
  • task #2154 pulp.server.async.tasks._release_resource
  • task #2386 pulp.server.controllers.repository.download_deferred
  • task #2388 pulp.server.controllers.repository.download_repo
  • not to be ported pulp.server.controllers.repository.queue_download_deferred
  • not to be ported pulp.server.controllers.repository.queue_publish
  • not to be ported pulp.server.db.reaper.queue_reap_expired_documents
  • task #2392 pulp.server.db.reaper.reap_expired_documents
  • task #2393 pulp.server.maintenance.monthly.monthly_maintenance
  • not to be ported pulp.server.maintenance.monthly.queue_monthly_maintenance
  • not to be ported pulp.server.managers.auth.permission.cud.grant
  • not to be ported pulp.server.managers.auth.permission.cud.revoke
  • not to be ported pulp.server.managers.auth.role.cud.add_permissions_to_role
  • not to be ported pulp.server.managers.auth.role.cud.add_user_to_role
  • not to be ported pulp.server.managers.auth.role.cud.create_role
  • not to be ported pulp.server.managers.auth.role.cud.delete_role
  • not to be ported pulp.server.managers.auth.role.cud.remove_permissions_from_role
  • not to be ported pulp.server.managers.auth.role.cud.remove_user_from_role
  • not to be ported pulp.server.managers.auth.role.cud.update_role
  • not to be ported pulp.server.managers.consumer.agent.delete_queue
  • task #2389 pulp.server.managers.consumer.applicability.batch_regenerate_applicability
  • task #2390 pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers
  • task #2391 pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos
  • not to be ported pulp.server.managers.consumer.bind.bind
  • not to be ported pulp.server.managers.consumer.bind.delete
  • not to be ported pulp.server.managers.consumer.bind.unbind
  • not to be ported pulp.server.managers.consumer.cud.register
  • not to be ported pulp.server.managers.consumer.cud.unregister
  • not to be ported pulp.server.managers.consumer.cud.update
  • not to be ported pulp.server.managers.consumer.group.cud.associate
  • not to be ported pulp.server.managers.consumer.group.cud.bind
  • not to be ported pulp.server.managers.consumer.group.cud.create_consumer_group
  • not to be ported pulp.server.managers.consumer.group.cud.delete_consumer_group
  • not to be ported pulp.server.managers.consumer.group.cud.unassociate
  • not to be ported pulp.server.managers.consumer.group.cud.unbind
  • not to be ported pulp.server.managers.consumer.group.cud.update_consumer_group
  • not to be ported pulp.server.managers.consumer.profile.create
  • not to be ported pulp.server.managers.consumer.profile.delete
  • not to be ported pulp.server.managers.consumer.profile.update
  • task #2394 pulp.server.managers.content.orphan.delete_all_orphans
  • task #2395 pulp.server.managers.content.orphan.delete_orphans_by_id
  • task #2396 pulp.server.managers.content.orphan.delete_orphans_by_type
  • task #2397 pulp.server.managers.content.upload.import_uploaded_unit
  • not to be ported pulp.server.managers.repo.group.cud.associate
  • not to be ported pulp.server.managers.repo.group.cud.create_and_configure_repo_group
  • not to be ported pulp.server.managers.repo.group.cud.delete_repo_group
  • not to be ported pulp.server.managers.repo.group.cud.unassociate
  • not to be ported pulp.server.managers.repo.group.cud.update_repo_group
  • not to be ported pulp.server.managers.repo.group.distributor.add_distributor
  • not to be ported pulp.server.managers.repo.group.distributor.remove_distributor
  • not to be ported pulp.server.managers.repo.group.distributor.update_distributor_config
  • not to be ported pulp.server.managers.repo.group.publish.publish
  • task #2400 pulp.server.managers.repo.importer.remove_importer
  • task #2401 pulp.server.managers.repo.importer.set_importer
  • task #2402 pulp.server.managers.repo.importer.update_importer_config
  • task #2398 pulp.server.managers.repo.publish.publish
  • task #2399 pulp.server.managers.repo.sync.sync
  • task #2403 pulp.server.managers.repo.unit_association.associate_from_repo
  • task #2404 pulp.server.managers.repo.unit_association.unassociate_by_criteria
  • not to be ported pulp.server.tasks.consumer.install_content
  • not to be ported pulp.server.tasks.consumer.uninstall_content
  • not to be ported pulp.server.tasks.consumer.update_content
  • task #2405 pulp.server.tasks.content.refresh_content_source
  • task #2406 pulp.server.tasks.content.refresh_content_sources
  • task #2407 pulp.server.tasks.repository.delete
  • task #2408 pulp.server.tasks.repository.distributor_delete
  • task #2409 pulp.server.tasks.repository.distributor_update
  • not to be ported pulp.server.tasks.repository.sync_with_auto_publish
Actions
Copy link
 #6

Updated by ttereshc about 8 years ago

  • Related to Refactor #2154: Convert task system to use django models added
Actions
Copy link
 #7

Updated by ttereshc about 8 years ago

  • Related to Task #2386: Convert celery task repository.download_deferred to Pulp 3 added
Actions
Copy link
 #8

Updated by mhrivnak about 8 years ago

All non-applicability consumer actions do not need porting. They can be done just through the REST API. They are tasks in pulp 2 because they often include performing actions on a consumer via gofer, which can take a while. Perhaps we were also doing resource locking for some, but that is definitely not needed with pulp 3.

Actions
Copy link
 #9

Updated by ttereshc about 8 years ago

  • Related to Task #2388: Convert celery task repository.download_repo to Pulp 3 added
Actions
Copy link
 #10

Updated by ttereshc about 8 years ago

  • Related to Task #2389: Convert celery task applicability.batch_regenerate_applicability to Pulp 3 added
Actions
Copy link
 #11

Updated by ttereshc about 8 years ago

  • Related to Task #2390: Convert celery task applicability.regenerate_applicability_for_consumers to Pulp 3 added
Actions
Copy link
 #12

Updated by ttereshc about 8 years ago

  • Related to Task #2391: Convert celery task applicability.regenerate_applicability_for_repos to Pulp 3 added
Actions
Copy link
 #13

Updated by ttereshc about 8 years ago

  • Groomed changed from No to Yes
Actions
Copy link
 #14

Updated by ttereshc about 8 years ago

  • Related to Task #2392: Convert celery task db.reaper.reap_expired_documents to Pulp 3 added
Actions
Copy link
 #15

Updated by ttereshc about 8 years ago

  • Related to Task #2393: Convert celery task monthly_maintenance to Pulp 3 added
Actions
Copy link
 #16

Updated by ttereshc about 8 years ago

  • Related to Task #2394: Convert celery task content.orphan.delete_all_orphans to Pulp 3 added
Actions
Copy link
 #17

Updated by ttereshc about 8 years ago

  • Related to Task #2395: Convert celery task content.orphan.delete_orphans_by_id to Pulp 3 added
Actions
Copy link
 #18

Updated by ttereshc about 8 years ago

  • Related to Story #2396: As a user, I can remove orphaned content and related artifacts by content type added
Actions
Copy link
 #19

Updated by ttereshc about 8 years ago

  • Related to Task #2397: Convert celery task content.upload.import_uploaded_unit to Pulp 3 added
Actions
Copy link
 #20

Updated by ttereshc about 8 years ago

  • Related to Task #2398: Convert celery task repo.publish to Pulp 3 added
Actions
Copy link
 #21

Updated by ttereshc about 8 years ago

  • Related to Task #2399: Convert celery task repo.sync to Pulp 3 added
Actions
Copy link
 #22

Updated by ttereshc about 8 years ago

  • Related to Task #2400: Convert celery task repo.importer.remove_importer to Pulp 3 added
Actions
Copy link
 #23

Updated by ttereshc about 8 years ago

  • Related to Task #2401: Convert celery task repo.importer.set_importer to Pulp 3 added
Actions
Copy link
 #24

Updated by ttereshc about 8 years ago

  • Related to Story #2402: As an API user, a call to update an Importer generates a Task added
Actions
Copy link
 #25

Updated by ttereshc about 8 years ago

  • Related to Task #2403: Convert celery task repo.unit_association.associate_from_repo to Pulp 3 added
Actions
Copy link
 #26

Updated by ttereshc about 8 years ago

  • Related to Task #2404: Convert celery task repo.unit_association.unassociate_by_criteria to Pulp 3 added
Actions
Copy link
 #27

Updated by ttereshc about 8 years ago

  • Related to Task #2405: Convert celery task content.refresh_content_source to Pulp 3 added
Actions
Copy link
 #28

Updated by ttereshc about 8 years ago

  • Related to Task #2406: Convert celery task content.refresh_content_sourceS to Pulp 3 added
Actions
Copy link
 #29

Updated by ttereshc about 8 years ago

  • Related to Task #2407: Convert celery task repository.delete to Pulp 3 added
Actions
Copy link
 #30

Updated by ttereshc about 8 years ago

  • Related to Task #2408: Convert celery task repository.distributor_delete to Pulp 3 added
Actions
Copy link
 #31

Updated by ttereshc about 8 years ago

  • Related to Story #2409: As an API user, a call to update an Publisher generates a Task added
Actions
Copy link
 #32

Updated by ttereshc about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
  • Sprint/Milestone set to 28

Someone, please, take a look at the comment 5 and check that I have not ruled out any celery task that we potentially need in Pulp 3.
If you think that I created a task which is not needed, just comment on it and close it.

Actions
Copy link
 #33

Updated by bmbouter about 8 years ago

Can all of the related issues have the 'Pulp 3' tag added. Other then that I think this issue can be closed and we can start commenting on the others.

Actions
Copy link
 #34

Updated by ttereshc about 8 years ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE

Thanks, I've added the tag.

Actions
Copy link
 #35

Updated by bmbouter about 8 years ago

Thank you ttereshc

Actions
Copy link
 #36

Updated by bmbouter almost 7 years ago

  • Sprint set to Sprint 10
Actions
Copy link
 #37

Updated by bmbouter almost 7 years ago

  • Sprint/Milestone deleted (28)
Actions
Copy link
 #38

Updated by daviddavis over 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions
Copy link
 #39

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions
Send by e-mail Copy link

Also available in: Atom PDF