Project

Profile

Help

Story #3442

As a user, I can clean up both orphaned content units and orphaned artifacts

Added by daviddavis over 1 year ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 34

Description

  • This would be a single action
  • I cannot specify the units specifically (all types).
  • I can follow the progress of all cleanups. (Cleanups are asynchronous.)
  • Call would be to DELETE /api/v3/orphans/.

Technical Design

Create a function that when called will:

@shared_task(base=UserFacingTask)
def orphan_cleanup():
    # This should contain all of the orphan code

Then make a celery task that will never actually run, but will be intercepted by queue_reserve_task(). Something like:

    if task.name == "orphan_cleanup":
        # loop forever while waiting for 0 reservations
        while True:
            if zero_reservations is True:
                try:
                    orphan_cleanup() # this automatically handles transitioning the task to RUNNING
                except:
                    orphan_cleanup.on_failure(...)  # we need to pass it the right args. This marks the task as failed.
                else:
                    orphan_cleanup.on_success(...)  # we need to pass it the right args. This marks the task as successful.
                return

Cancellation

To ensure that task cancellation work as expected we should have the _queue_release_resource receive the task ID typically specified by the inner_task_id. This should be safe because _queue_release_resource has no side effects on the Task data because it does not inherit from UserFacingTask

To accomplish ^ we need to have a conditional path like:

if task.name == "orphan_cleanup":
        _queue_reserved_task.apply_async(task_id=inner_task_id, args=(task_name, inner_task_id, list(resources), args,
                                               kwargs, options),
                                         queue=RESOURCE_MANAGER_QUEUE)
else:
        _queue_reserved_task.apply_async(args=(task_name, inner_task_id, list(resources), args,
                                               kwargs, options),
                                         queue=RESOURCE_MANAGER_QUEUE)


Related issues

Related to Pulp - Issue #3445: Remove the ability for users to delete a single content unit. MODIFIED Actions
Related to Pulp - Issue #3508: Artifact files aren't getting deleted along with artifacts MODIFIED Actions
Related to Pulp - Story #3542: As a plugin writer, I can create a task that waits until there are no reservations NEW Actions
Related to Pulp - Story #2484: As a pulp 3.0 user, orphan content delete reports how many units were deleted CLOSED - CURRENTRELEASE Actions
Related to Pulp - Test #4663: Test remove orphans endpoint CLOSED - DUPLICATE Actions

Associated revisions

Revision 3eb5b1a2 View on GitHub
Added by daviddavis over 1 year ago

As a user, I can cleanup orphan content and artifacts

fixes #3442
https://pulp.plan.io/issues/3442

Revision 3eb5b1a2 View on GitHub
Added by daviddavis over 1 year ago

As a user, I can cleanup orphan content and artifacts

fixes #3442
https://pulp.plan.io/issues/3442

Revision 3eb5b1a2 View on GitHub
Added by daviddavis over 1 year ago

As a user, I can cleanup orphan content and artifacts

fixes #3442
https://pulp.plan.io/issues/3442

History

#1 Updated by daviddavis over 1 year ago

I think we just need to figure out the endpoint to get this groomed.

#2 Updated by daviddavis over 1 year ago

  • Tracker changed from Issue to Story
  • % Done set to 0
  • Tags Pulp 3, Pulp 3 MVP added

#3 Updated by jortel@redhat.com over 1 year ago

Even though the MVP specifies delete orphaned content by type, I really doubt that is a real use.

#4 Updated by daviddavis over 1 year ago

  • Description updated (diff)

Updating proposal after discussion with @jortel and @milan on IRC.

#5 Updated by daviddavis over 1 year ago

A couple other ideas we came up with:

  • Users can select between removing just content units or just artifacts
  • GET /api/v3/orphans = list orphans

We agreed to leave these off for later.

#6 Updated by jortel@redhat.com over 1 year ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#7 Updated by daviddavis over 1 year ago

  • Groomed changed from Yes to No
  • Sprint Candidate changed from Yes to No

I thought we could implement this without the tasking system but as @bmbouter pointed out, this would fail when (for example) an importer creates a unit and then adds it to a repo version. If the orphan cleanup runs between the two steps, the unit gets deleted. One solution would be to have an orphan cleanup task which checks to make sure there are 0 reserved tasks before running.

Setting to ungroomed until we have a better technical design.

#8 Updated by bmbouter over 1 year ago

  • Description updated (diff)

Adding a technical design proposal in the hopes that we can groom it.

#9 Updated by amacdona@redhat.com over 1 year ago

  • Description updated (diff)

#10 Updated by daviddavis over 1 year ago

  • Related to Issue #3445: Remove the ability for users to delete a single content unit. added

#11 Updated by bmbouter over 1 year ago

  • Description updated (diff)

After talking w/ @jortel online, we wanted to clarify two aspects of the design:

1. That it would use try/except/else and
2. That we can use __call__ and on_success/on_failure to handle the task state transitions.

Also as an aside, it's important that orphan_cleanup be an actual Celery task even though it will always be called synchronously because that allows it's __call__ to handle the task state transitions.

#12 Updated by bmbouter over 1 year ago

Also note that by having only the orphan cleanup code in the celery task itself, any time the resource manager is waiting for the reservations count to go to 0 will have the task shown as 'waiting'. I think this is good because the user could take an action to cancel the work that is currently running. Also the task runtime of orphan cleanup will be the actual code runtime that will not include the waiting time which is consistent with other tasks.

#13 Updated by jortel@redhat.com over 1 year ago

The changes good. Although, using this special name seems brittle. Perhaps we can add an attribute to the task that stipulates that it needs to be run with zero reservations instead.

The if task.name == "orphan_cleanup":

The orphan clean up task could be long running. I suspect it cannot be canceled, right? What happens if a user tries to cancel the task?

#14 Updated by bmbouter over 1 year ago

  • Description updated (diff)

#15 Updated by jortel@redhat.com over 1 year ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#16 Updated by jortel@redhat.com over 1 year ago

  • Sprint set to Sprint 34

#17 Updated by daviddavis over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#18 Updated by daviddavis over 1 year ago

  • Related to Issue #3508: Artifact files aren't getting deleted along with artifacts added

#19 Updated by daviddavis over 1 year ago

  • Smash Test set to 914

#20 Updated by daviddavis over 1 year ago

  • Related to Story #3542: As a plugin writer, I can create a task that waits until there are no reservations added

#21 Updated by daviddavis over 1 year ago

Per comment 13, I opened a new issue to handle making this functionality around waiting for no reservations more generic so plugin writers, etc can use it:

https://pulp.plan.io/issues/3542

This should also take care of @jortel's concern about name being being too brittle.

#22 Updated by daviddavis over 1 year ago

  • Status changed from ASSIGNED to POST

#23 Updated by daviddavis over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#24 Updated by dalley over 1 year ago

  • Related to Story #2484: As a pulp 3.0 user, orphan content delete reports how many units were deleted added

#25 Updated by dkliban@redhat.com over 1 year ago

  • Sprint/Milestone set to 3.0

#26 Updated by kersom 6 months ago

  • Related to Test #4663: Test remove orphans endpoint added

#27 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3, Pulp 3 MVP)

Please register to edit this issue

Also available in: Atom PDF