Story #3442
closedAs a user, I can clean up both orphaned content units and orphaned artifacts
100%
Description
- This would be a single action
- I cannot specify the units specifically (all types).
- I can follow the progress of all cleanups. (Cleanups are asynchronous.)
- Call would be to DELETE /api/v3/orphans/.
Technical Design¶
Create a function that when called will:
@shared_task(base=UserFacingTask)
def orphan_cleanup():
# This should contain all of the orphan code
Then make a celery task that will never actually run, but will be intercepted by queue_reserve_task()
. Something like:
if task.name == "orphan_cleanup":
# loop forever while waiting for 0 reservations
while True:
if zero_reservations is True:
try:
orphan_cleanup() # this automatically handles transitioning the task to RUNNING
except:
orphan_cleanup.on_failure(...) # we need to pass it the right args. This marks the task as failed.
else:
orphan_cleanup.on_success(...) # we need to pass it the right args. This marks the task as successful.
return
Cancellation¶
To ensure that task cancellation work as expected we should have the _queue_release_resource receive the task ID typically specified by the inner_task_id
. This should be safe because _queue_release_resource has no side effects on the Task data because it does not inherit from UserFacingTask
To accomplish ^ we need to have a conditional path like:
if task.name == "orphan_cleanup":
_queue_reserved_task.apply_async(task_id=inner_task_id, args=(task_name, inner_task_id, list(resources), args,
kwargs, options),
queue=RESOURCE_MANAGER_QUEUE)
else:
_queue_reserved_task.apply_async(args=(task_name, inner_task_id, list(resources), args,
kwargs, options),
queue=RESOURCE_MANAGER_QUEUE)
Related issues
Updated by daviddavis over 6 years ago
I think we just need to figure out the endpoint to get this groomed.
Updated by daviddavis over 6 years ago
- Tracker changed from Issue to Story
- % Done set to 0
- Tags Pulp 3, Pulp 3 MVP added
Updated by jortel@redhat.com over 6 years ago
Even though the MVP specifies delete orphaned content by type, I really doubt that is a real use.
Updated by daviddavis over 6 years ago
- Description updated (diff)
Updating proposal after discussion with @jortel and milan on IRC.
Updated by daviddavis over 6 years ago
A couple other ideas we came up with:
- Users can select between removing just content units or just artifacts
- GET /api/v3/orphans = list orphans
We agreed to leave these off for later.
Updated by jortel@redhat.com over 6 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by daviddavis over 6 years ago
- Groomed changed from Yes to No
- Sprint Candidate changed from Yes to No
I thought we could implement this without the tasking system but as bmbouter pointed out, this would fail when (for example) an importer creates a unit and then adds it to a repo version. If the orphan cleanup runs between the two steps, the unit gets deleted. One solution would be to have an orphan cleanup task which checks to make sure there are 0 reserved tasks before running.
Setting to ungroomed until we have a better technical design.
Updated by bmbouter over 6 years ago
- Description updated (diff)
Adding a technical design proposal in the hopes that we can groom it.
Updated by daviddavis over 6 years ago
- Related to Issue #3445: Remove the ability for users to delete a single content unit. added
Updated by bmbouter over 6 years ago
- Description updated (diff)
After talking w/ @jortel online, we wanted to clarify two aspects of the design:
1. That it would use try/except/else and
2. That we can use __call__
and on_success/on_failure
to handle the task state transitions.
Also as an aside, it's important that orphan_cleanup be an actual Celery task even though it will always be called synchronously because that allows it's __call__
to handle the task state transitions.
Updated by bmbouter over 6 years ago
Also note that by having only the orphan cleanup code in the celery task itself, any time the resource manager is waiting for the reservations count to go to 0 will have the task shown as 'waiting'. I think this is good because the user could take an action to cancel the work that is currently running. Also the task runtime of orphan cleanup will be the actual code runtime that will not include the waiting time which is consistent with other tasks.
Updated by jortel@redhat.com over 6 years ago
The changes good. Although, using this special name seems brittle. Perhaps we can add an attribute to the task that stipulates that it needs to be run with zero reservations instead.
The if task.name == "orphan_cleanup":
The orphan clean up task could be long running. I suspect it cannot be canceled, right? What happens if a user tries to cancel the task?
Updated by jortel@redhat.com over 6 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by daviddavis over 6 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to daviddavis
Updated by daviddavis over 6 years ago
- Related to Issue #3508: Artifact files aren't getting deleted along with artifacts added
Updated by daviddavis over 6 years ago
- Related to Story #3542: As a plugin writer, I can create a task that waits until there are no reservations added
Updated by daviddavis over 6 years ago
Per comment 13, I opened a new issue to handle making this functionality around waiting for no reservations more generic so plugin writers, etc can use it:
https://pulp.plan.io/issues/3542
This should also take care of @jortel's concern about name being being too brittle.
Updated by daviddavis over 6 years ago
- Status changed from ASSIGNED to POST
Added by daviddavis over 6 years ago
Added by daviddavis over 6 years ago
Revision 3eb5b1a2 | View on GitHub
As a user, I can cleanup orphan content and artifacts
Updated by daviddavis over 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|3eb5b1a273ba96c4bd3ab6c0f26e42b002633b66.
Updated by dalley over 6 years ago
- Related to Story #2484: As a pulp 3.0 user, orphan content delete reports how many units were deleted added
Updated by kersom over 5 years ago
- Related to Test #4663: Test remove orphans endpoint added
Updated by bmbouter almost 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
As a user, I can cleanup orphan content and artifacts
fixes #3442 https://pulp.plan.io/issues/3442