As a user, I can clean up both orphaned content units and orphaned artifacts
- This would be a single action
- I cannot specify the units specifically (all types).
- I can follow the progress of all cleanups. (Cleanups are asynchronous.)
- Call would be to DELETE /api/v3/orphans/.
Create a function that when called will:
@shared_task(base=UserFacingTask) def orphan_cleanup(): # This should contain all of the orphan code
Then make a celery task that will never actually run, but will be intercepted by
queue_reserve_task(). Something like:
if task.name == "orphan_cleanup": # loop forever while waiting for 0 reservations while True: if zero_reservations is True: try: orphan_cleanup() # this automatically handles transitioning the task to RUNNING except: orphan_cleanup.on_failure(...) # we need to pass it the right args. This marks the task as failed. else: orphan_cleanup.on_success(...) # we need to pass it the right args. This marks the task as successful. return
To ensure that task cancellation work as expected we should have the _queue_release_resource receive the task ID typically specified by the
inner_task_id. This should be safe because _queue_release_resource has no side effects on the Task data because it does not inherit from UserFacingTask
To accomplish ^ we need to have a conditional path like:
if task.name == "orphan_cleanup": _queue_reserved_task.apply_async(task_id=inner_task_id, args=(task_name, inner_task_id, list(resources), args, kwargs, options), queue=RESOURCE_MANAGER_QUEUE) else: _queue_reserved_task.apply_async(args=(task_name, inner_task_id, list(resources), args, kwargs, options), queue=RESOURCE_MANAGER_QUEUE)
#7 Updated by daviddavis almost 3 years ago
- Groomed changed from Yes to No
- Sprint Candidate changed from Yes to No
I thought we could implement this without the tasking system but as bmbouter pointed out, this would fail when (for example) an importer creates a unit and then adds it to a repo version. If the orphan cleanup runs between the two steps, the unit gets deleted. One solution would be to have an orphan cleanup task which checks to make sure there are 0 reserved tasks before running.
Setting to ungroomed until we have a better technical design.
#11 Updated by bmbouter almost 3 years ago
- Description updated (diff)
After talking w/ @jortel online, we wanted to clarify two aspects of the design:
1. That it would use try/except/else and
2. That we can use
on_success/on_failure to handle the task state transitions.
Also as an aside, it's important that orphan_cleanup be an actual Celery task even though it will always be called synchronously because that allows it's
__call__ to handle the task state transitions.
#12 Updated by bmbouter almost 3 years ago
Also note that by having only the orphan cleanup code in the celery task itself, any time the resource manager is waiting for the reservations count to go to 0 will have the task shown as 'waiting'. I think this is good because the user could take an action to cancel the work that is currently running. Also the task runtime of orphan cleanup will be the actual code runtime that will not include the waiting time which is consistent with other tasks.
#13 Updated by email@example.com almost 3 years ago
The changes good. Although, using this special name seems brittle. Perhaps we can add an attribute to the task that stipulates that it needs to be run with zero reservations instead.
The if task.name == "orphan_cleanup":
The orphan clean up task could be long running. I suspect it cannot be canceled, right? What happens if a user tries to cancel the task?
#21 Updated by daviddavis almost 3 years ago
Per comment 13, I opened a new issue to handle making this functionality around waiting for no reservations more generic so plugin writers, etc can use it:
This should also take care of @jortel's concern about name being being too brittle.
Please register to edit this issue