Project

Profile

Help

Task #2380

Create a redmine task for each 2.y celery task to be converted to 3.0

Added by bmbouter about 3 years ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 10

Description

We have roughly 20-30 celery tasks in Pulp 2.y. We need to have a redmine task created for each one tracking the conversion of that celery task to Pulp3.

To identify the list of celery tasks in 2.y I recommend inspecting the celery registry of a 2.y dev install. You can inspect the registry using a technique similar to what this0 unit test does.

Each task needs to be moved out of the pulp.server and into somewhere. We should discuss on the issues themselves where each one should go. Also on Pulp3 all of the tasks are expected to be imported into pulp.tasking.registry which is where celery will discover them. Note that it's probably not a good idea to have the task code itself live in pulp.tasking.registry so where to house that code is something we should discuss.

The details of what the Pulp3 behaviors should be for that task can be discussed on each issue created. There are a few that will be deleted. Let's make issues for all the 2.y tasks and then we can identify ones to close on the individual issues.

[0]: https://github.com/bmbouter/pulp/blob/1fe832ce1d320129c680755f38cafae26f5159d0/server/test/unit/server/async/test_tasks.py#L1053-L1053


Related issues

Related to Pulp - Refactor #2154: Convert task system to use django models MODIFIED Actions
Related to Pulp - Task #2386: Convert celery task repository.download_deferred to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2388: Convert celery task repository.download_repo to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2389: Convert celery task applicability.batch_regenerate_applicability to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2390: Convert celery task applicability.regenerate_applicability_for_consumers to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2391: Convert celery task applicability.regenerate_applicability_for_repos to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2392: Convert celery task db.reaper.reap_expired_documents to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2393: Convert celery task monthly_maintenance to Pulp 3 CLOSED - NOTABUG Actions
Related to Pulp - Task #2394: Convert celery task content.orphan.delete_all_orphans to Pulp 3 MODIFIED Actions
Related to Pulp - Task #2395: Convert celery task content.orphan.delete_orphans_by_id to Pulp 3 NEW Actions
Related to Pulp - Story #2396: As a user, I can remove orphaned content and related artifacts by content type NEW Actions
Related to Pulp - Task #2397: Convert celery task content.upload.import_uploaded_unit to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2398: Convert celery task repo.publish to Pulp 3 MODIFIED Actions
Related to Pulp - Task #2399: Convert celery task repo.sync to Pulp 3 MODIFIED Actions
Related to Pulp - Task #2400: Convert celery task repo.importer.remove_importer to Pulp 3 MODIFIED Actions
Related to Pulp - Task #2401: Convert celery task repo.importer.set_importer to Pulp 3 CLOSED - DUPLICATE Actions
Related to Pulp - Story #2402: As an API user, a call to update an Importer generates a Task MODIFIED Actions
Related to Pulp - Task #2403: Convert celery task repo.unit_association.associate_from_repo to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2404: Convert celery task repo.unit_association.unassociate_by_criteria to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2405: Convert celery task content.refresh_content_source to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2406: Convert celery task content.refresh_content_sourceS to Pulp 3 CLOSED - WONTFIX Actions
Related to Pulp - Task #2407: Convert celery task repository.delete to Pulp 3 MODIFIED Actions
Related to Pulp - Task #2408: Convert celery task repository.distributor_delete to Pulp 3 MODIFIED Actions
Related to Pulp - Story #2409: As an API user, a call to update an Publisher generates a Task MODIFIED Actions

History

#1 Updated by mhrivnak about 3 years ago

$ sudo -u apache celery inspect registered -A pulp.server.async.app
-> reserved_resource_worker-0@dev: OK
    * pulp.server.async.tasks.PulpTask
    * pulp.server.async.tasks.Task
    * pulp.server.async.tasks._queue_reserved_task
    * pulp.server.async.tasks._release_resource
    * pulp.server.controllers.repository.download_deferred
    * pulp.server.controllers.repository.download_repo
    * pulp.server.controllers.repository.queue_download_deferred
    * pulp.server.controllers.repository.queue_publish
    * pulp.server.db.reaper.queue_reap_expired_documents
    * pulp.server.db.reaper.reap_expired_documents
    * pulp.server.maintenance.monthly.monthly_maintenance
    * pulp.server.maintenance.monthly.queue_monthly_maintenance
    * pulp.server.managers.auth.permission.cud.grant
    * pulp.server.managers.auth.permission.cud.revoke
    * pulp.server.managers.auth.role.cud.add_permissions_to_role
    * pulp.server.managers.auth.role.cud.add_user_to_role
    * pulp.server.managers.auth.role.cud.create_role
    * pulp.server.managers.auth.role.cud.delete_role
    * pulp.server.managers.auth.role.cud.remove_permissions_from_role
    * pulp.server.managers.auth.role.cud.remove_user_from_role
    * pulp.server.managers.auth.role.cud.update_role
    * pulp.server.managers.consumer.agent.delete_queue
    * pulp.server.managers.consumer.applicability.batch_regenerate_applicability
    * pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers
    * pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos
    * pulp.server.managers.consumer.bind.bind
    * pulp.server.managers.consumer.bind.delete
    * pulp.server.managers.consumer.bind.unbind
    * pulp.server.managers.consumer.cud.register
    * pulp.server.managers.consumer.cud.unregister
    * pulp.server.managers.consumer.cud.update
    * pulp.server.managers.consumer.group.cud.associate
    * pulp.server.managers.consumer.group.cud.bind
    * pulp.server.managers.consumer.group.cud.create_consumer_group
    * pulp.server.managers.consumer.group.cud.delete_consumer_group
    * pulp.server.managers.consumer.group.cud.unassociate
    * pulp.server.managers.consumer.group.cud.unbind
    * pulp.server.managers.consumer.group.cud.update_consumer_group
    * pulp.server.managers.consumer.profile.create
    * pulp.server.managers.consumer.profile.delete
    * pulp.server.managers.consumer.profile.update
    * pulp.server.managers.content.orphan.delete_all_orphans
    * pulp.server.managers.content.orphan.delete_orphans_by_id
    * pulp.server.managers.content.orphan.delete_orphans_by_type
    * pulp.server.managers.content.upload.import_uploaded_unit
    * pulp.server.managers.repo.group.cud.associate
    * pulp.server.managers.repo.group.cud.create_and_configure_repo_group
    * pulp.server.managers.repo.group.cud.delete_repo_group
    * pulp.server.managers.repo.group.cud.unassociate
    * pulp.server.managers.repo.group.cud.update_repo_group
    * pulp.server.managers.repo.group.distributor.add_distributor
    * pulp.server.managers.repo.group.distributor.remove_distributor
    * pulp.server.managers.repo.group.distributor.update_distributor_config
    * pulp.server.managers.repo.group.publish.publish
    * pulp.server.managers.repo.importer.remove_importer
    * pulp.server.managers.repo.importer.set_importer
    * pulp.server.managers.repo.importer.update_importer_config
    * pulp.server.managers.repo.publish.publish
    * pulp.server.managers.repo.sync.sync
    * pulp.server.managers.repo.unit_association.associate_from_repo
    * pulp.server.managers.repo.unit_association.unassociate_by_criteria
    * pulp.server.tasks.consumer.install_content
    * pulp.server.tasks.consumer.uninstall_content
    * pulp.server.tasks.consumer.update_content
    * pulp.server.tasks.content.refresh_content_source
    * pulp.server.tasks.content.refresh_content_sources
    * pulp.server.tasks.repository.delete
    * pulp.server.tasks.repository.distributor_delete
    * pulp.server.tasks.repository.distributor_update
    * pulp.server.tasks.repository.sync_with_auto_publish

#2 Updated by mhrivnak about 3 years ago

I think these can be ignored, and not ported to pulp 3:
  1. auth tasks
  2. consumer tasks that aren't related to applicability
  3. repo group tasks
  4. Any task that was only used by the scheduler to queue other user-facing tasks. Example: queue_publish (also, I'm not sure why "queue_sync_with_auto_publish" is missing from this list)

#3 Updated by bmbouter about 3 years ago

Thanks @mhrivnak. Yes all of those things should not be ported.

Also, these 4 are already ported so they can be ignored.

    * pulp.server.async.tasks.PulpTask
    * pulp.server.async.tasks.Task
    * pulp.server.async.tasks._queue_reserved_task
    * pulp.server.async.tasks._release_resource

#4 Updated by ttereshc about 3 years ago

mhrivnak wrote:

  1. Any task that was only used by the scheduler to queue other user-facing tasks. Example: queue_publish (also, I'm not sure why "queue_sync_with_auto_publish" is missing from this list)

It is there, the name is without "queue_" for some reason, just sync_with_auto_publish

#5 Updated by ttereshc about 3 years ago

This comment is just to track what celery tasks have been covered so far. Feel free to edit it accordingly.

  • task #2154 pulp.server.async.tasks.PulpTask
  • task #2154 pulp.server.async.tasks.Task
  • task #2154 pulp.server.async.tasks._queue_reserved_task
  • task #2154 pulp.server.async.tasks._release_resource
  • task #2386 pulp.server.controllers.repository.download_deferred
  • task #2388 pulp.server.controllers.repository.download_repo
  • not to be ported pulp.server.controllers.repository.queue_download_deferred
  • not to be ported pulp.server.controllers.repository.queue_publish
  • not to be ported pulp.server.db.reaper.queue_reap_expired_documents
  • task #2392 pulp.server.db.reaper.reap_expired_documents
  • task #2393 pulp.server.maintenance.monthly.monthly_maintenance
  • not to be ported pulp.server.maintenance.monthly.queue_monthly_maintenance
  • not to be ported pulp.server.managers.auth.permission.cud.grant
  • not to be ported pulp.server.managers.auth.permission.cud.revoke
  • not to be ported pulp.server.managers.auth.role.cud.add_permissions_to_role
  • not to be ported pulp.server.managers.auth.role.cud.add_user_to_role
  • not to be ported pulp.server.managers.auth.role.cud.create_role
  • not to be ported pulp.server.managers.auth.role.cud.delete_role
  • not to be ported pulp.server.managers.auth.role.cud.remove_permissions_from_role
  • not to be ported pulp.server.managers.auth.role.cud.remove_user_from_role
  • not to be ported pulp.server.managers.auth.role.cud.update_role
  • not to be ported pulp.server.managers.consumer.agent.delete_queue
  • task #2389 pulp.server.managers.consumer.applicability.batch_regenerate_applicability
  • task #2390 pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers
  • task #2391 pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos
  • not to be ported pulp.server.managers.consumer.bind.bind
  • not to be ported pulp.server.managers.consumer.bind.delete
  • not to be ported pulp.server.managers.consumer.bind.unbind
  • not to be ported pulp.server.managers.consumer.cud.register
  • not to be ported pulp.server.managers.consumer.cud.unregister
  • not to be ported pulp.server.managers.consumer.cud.update
  • not to be ported pulp.server.managers.consumer.group.cud.associate
  • not to be ported pulp.server.managers.consumer.group.cud.bind
  • not to be ported pulp.server.managers.consumer.group.cud.create_consumer_group
  • not to be ported pulp.server.managers.consumer.group.cud.delete_consumer_group
  • not to be ported pulp.server.managers.consumer.group.cud.unassociate
  • not to be ported pulp.server.managers.consumer.group.cud.unbind
  • not to be ported pulp.server.managers.consumer.group.cud.update_consumer_group
  • not to be ported pulp.server.managers.consumer.profile.create
  • not to be ported pulp.server.managers.consumer.profile.delete
  • not to be ported pulp.server.managers.consumer.profile.update
  • task #2394 pulp.server.managers.content.orphan.delete_all_orphans
  • task #2395 pulp.server.managers.content.orphan.delete_orphans_by_id
  • task #2396 pulp.server.managers.content.orphan.delete_orphans_by_type
  • task #2397 pulp.server.managers.content.upload.import_uploaded_unit
  • not to be ported pulp.server.managers.repo.group.cud.associate
  • not to be ported pulp.server.managers.repo.group.cud.create_and_configure_repo_group
  • not to be ported pulp.server.managers.repo.group.cud.delete_repo_group
  • not to be ported pulp.server.managers.repo.group.cud.unassociate
  • not to be ported pulp.server.managers.repo.group.cud.update_repo_group
  • not to be ported pulp.server.managers.repo.group.distributor.add_distributor
  • not to be ported pulp.server.managers.repo.group.distributor.remove_distributor
  • not to be ported pulp.server.managers.repo.group.distributor.update_distributor_config
  • not to be ported pulp.server.managers.repo.group.publish.publish
  • task #2400 pulp.server.managers.repo.importer.remove_importer
  • task #2401 pulp.server.managers.repo.importer.set_importer
  • task #2402 pulp.server.managers.repo.importer.update_importer_config
  • task #2398 pulp.server.managers.repo.publish.publish
  • task #2399 pulp.server.managers.repo.sync.sync
  • task #2403 pulp.server.managers.repo.unit_association.associate_from_repo
  • task #2404 pulp.server.managers.repo.unit_association.unassociate_by_criteria
  • not to be ported pulp.server.tasks.consumer.install_content
  • not to be ported pulp.server.tasks.consumer.uninstall_content
  • not to be ported pulp.server.tasks.consumer.update_content
  • task #2405 pulp.server.tasks.content.refresh_content_source
  • task #2406 pulp.server.tasks.content.refresh_content_sources
  • task #2407 pulp.server.tasks.repository.delete
  • task #2408 pulp.server.tasks.repository.distributor_delete
  • task #2409 pulp.server.tasks.repository.distributor_update
  • not to be ported pulp.server.tasks.repository.sync_with_auto_publish

#6 Updated by ttereshc about 3 years ago

  • Related to Refactor #2154: Convert task system to use django models added

#7 Updated by ttereshc about 3 years ago

  • Related to Task #2386: Convert celery task repository.download_deferred to Pulp 3 added

#8 Updated by mhrivnak about 3 years ago

All non-applicability consumer actions do not need porting. They can be done just through the REST API. They are tasks in pulp 2 because they often include performing actions on a consumer via gofer, which can take a while. Perhaps we were also doing resource locking for some, but that is definitely not needed with pulp 3.

#9 Updated by ttereshc about 3 years ago

  • Related to Task #2388: Convert celery task repository.download_repo to Pulp 3 added

#10 Updated by ttereshc about 3 years ago

  • Related to Task #2389: Convert celery task applicability.batch_regenerate_applicability to Pulp 3 added

#11 Updated by ttereshc about 3 years ago

  • Related to Task #2390: Convert celery task applicability.regenerate_applicability_for_consumers to Pulp 3 added

#12 Updated by ttereshc about 3 years ago

  • Related to Task #2391: Convert celery task applicability.regenerate_applicability_for_repos to Pulp 3 added

#13 Updated by ttereshc about 3 years ago

  • Groomed changed from No to Yes

#14 Updated by ttereshc about 3 years ago

  • Related to Task #2392: Convert celery task db.reaper.reap_expired_documents to Pulp 3 added

#15 Updated by ttereshc about 3 years ago

  • Related to Task #2393: Convert celery task monthly_maintenance to Pulp 3 added

#16 Updated by ttereshc about 3 years ago

  • Related to Task #2394: Convert celery task content.orphan.delete_all_orphans to Pulp 3 added

#17 Updated by ttereshc about 3 years ago

  • Related to Task #2395: Convert celery task content.orphan.delete_orphans_by_id to Pulp 3 added

#18 Updated by ttereshc about 3 years ago

  • Related to Story #2396: As a user, I can remove orphaned content and related artifacts by content type added

#19 Updated by ttereshc about 3 years ago

  • Related to Task #2397: Convert celery task content.upload.import_uploaded_unit to Pulp 3 added

#20 Updated by ttereshc about 3 years ago

  • Related to Task #2398: Convert celery task repo.publish to Pulp 3 added

#21 Updated by ttereshc about 3 years ago

  • Related to Task #2399: Convert celery task repo.sync to Pulp 3 added

#22 Updated by ttereshc about 3 years ago

  • Related to Task #2400: Convert celery task repo.importer.remove_importer to Pulp 3 added

#23 Updated by ttereshc about 3 years ago

  • Related to Task #2401: Convert celery task repo.importer.set_importer to Pulp 3 added

#24 Updated by ttereshc about 3 years ago

  • Related to Story #2402: As an API user, a call to update an Importer generates a Task added

#25 Updated by ttereshc about 3 years ago

  • Related to Task #2403: Convert celery task repo.unit_association.associate_from_repo to Pulp 3 added

#26 Updated by ttereshc about 3 years ago

  • Related to Task #2404: Convert celery task repo.unit_association.unassociate_by_criteria to Pulp 3 added

#27 Updated by ttereshc about 3 years ago

  • Related to Task #2405: Convert celery task content.refresh_content_source to Pulp 3 added

#28 Updated by ttereshc about 3 years ago

  • Related to Task #2406: Convert celery task content.refresh_content_sourceS to Pulp 3 added

#29 Updated by ttereshc about 3 years ago

  • Related to Task #2407: Convert celery task repository.delete to Pulp 3 added

#30 Updated by ttereshc about 3 years ago

  • Related to Task #2408: Convert celery task repository.distributor_delete to Pulp 3 added

#31 Updated by ttereshc about 3 years ago

  • Related to Story #2409: As an API user, a call to update an Publisher generates a Task added

#32 Updated by ttereshc about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
  • Sprint/Milestone set to 28

Someone, please, take a look at the comment 5 and check that I have not ruled out any celery task that we potentially need in Pulp 3.
If you think that I created a task which is not needed, just comment on it and close it.

#33 Updated by bmbouter about 3 years ago

Can all of the related issues have the 'Pulp 3' tag added. Other then that I think this issue can be closed and we can start commenting on the others.

#34 Updated by ttereshc about 3 years ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE

Thanks, I've added the tag.

#35 Updated by bmbouter about 3 years ago

Thank you @ttereshc

#36 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 10

#37 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (28)

#38 Updated by daviddavis 7 months ago

  • Sprint/Milestone set to 3.0

#39 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF