Project

Profile

Help

Story #3442

closed

As a user, I can clean up both orphaned content units and orphaned artifacts

Added by daviddavis about 6 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 34
Quarter:

Description

  • This would be a single action
  • I cannot specify the units specifically (all types).
  • I can follow the progress of all cleanups. (Cleanups are asynchronous.)
  • Call would be to DELETE /api/v3/orphans/.

Technical Design

Create a function that when called will:

@shared_task(base=UserFacingTask)
def orphan_cleanup():
    # This should contain all of the orphan code

Then make a celery task that will never actually run, but will be intercepted by queue_reserve_task(). Something like:

    if task.name == "orphan_cleanup":
        # loop forever while waiting for 0 reservations
        while True:
            if zero_reservations is True:
                try:
                    orphan_cleanup() # this automatically handles transitioning the task to RUNNING
                except:
                    orphan_cleanup.on_failure(...)  # we need to pass it the right args. This marks the task as failed.
                else:
                    orphan_cleanup.on_success(...)  # we need to pass it the right args. This marks the task as successful.
                return

Cancellation

To ensure that task cancellation work as expected we should have the _queue_release_resource receive the task ID typically specified by the inner_task_id. This should be safe because _queue_release_resource has no side effects on the Task data because it does not inherit from UserFacingTask

To accomplish ^ we need to have a conditional path like:

if task.name == "orphan_cleanup":
        _queue_reserved_task.apply_async(task_id=inner_task_id, args=(task_name, inner_task_id, list(resources), args,
                                               kwargs, options),
                                         queue=RESOURCE_MANAGER_QUEUE)
else:
        _queue_reserved_task.apply_async(args=(task_name, inner_task_id, list(resources), args,
                                               kwargs, options),
                                         queue=RESOURCE_MANAGER_QUEUE)

Related issues

Related to Pulp - Issue #3445: Remove the ability for users to delete a single content unit.CLOSED - CURRENTRELEASEdalleyActions
Related to Pulp - Issue #3508: Artifact files aren't getting deleted along with artifactsCLOSED - CURRENTRELEASEdaviddavisActions
Related to Pulp - Story #3542: As a plugin writer, I can create a task that waits until there are no reservationsCLOSED - WONTFIX

Actions
Related to Pulp - Story #2484: As a pulp 3.0 user, orphan content delete reports how many units were deletedCLOSED - CURRENTRELEASE

Actions
Related to Pulp - Test #4663: Test remove orphans endpointCLOSED - DUPLICATEkersomActions
Actions #1

Updated by daviddavis about 6 years ago

I think we just need to figure out the endpoint to get this groomed.

Actions #2

Updated by daviddavis about 6 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0
  • Tags Pulp 3, Pulp 3 MVP added
Actions #3

Updated by jortel@redhat.com about 6 years ago

Even though the MVP specifies delete orphaned content by type, I really doubt that is a real use.

Actions #4

Updated by daviddavis about 6 years ago

  • Description updated (diff)

Updating proposal after discussion with @jortel and milan on IRC.

Actions #5

Updated by daviddavis about 6 years ago

A couple other ideas we came up with:

  • Users can select between removing just content units or just artifacts
  • GET /api/v3/orphans = list orphans

We agreed to leave these off for later.

Actions #6

Updated by jortel@redhat.com about 6 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #7

Updated by daviddavis about 6 years ago

  • Groomed changed from Yes to No
  • Sprint Candidate changed from Yes to No

I thought we could implement this without the tasking system but as bmbouter pointed out, this would fail when (for example) an importer creates a unit and then adds it to a repo version. If the orphan cleanup runs between the two steps, the unit gets deleted. One solution would be to have an orphan cleanup task which checks to make sure there are 0 reserved tasks before running.

Setting to ungroomed until we have a better technical design.

Actions #8

Updated by bmbouter about 6 years ago

  • Description updated (diff)

Adding a technical design proposal in the hopes that we can groom it.

Actions #9

Updated by amacdona@redhat.com about 6 years ago

  • Description updated (diff)
Actions #10

Updated by daviddavis about 6 years ago

  • Related to Issue #3445: Remove the ability for users to delete a single content unit. added
Actions #11

Updated by bmbouter about 6 years ago

  • Description updated (diff)

After talking w/ @jortel online, we wanted to clarify two aspects of the design:

1. That it would use try/except/else and
2. That we can use __call__ and on_success/on_failure to handle the task state transitions.

Also as an aside, it's important that orphan_cleanup be an actual Celery task even though it will always be called synchronously because that allows it's __call__ to handle the task state transitions.

Actions #12

Updated by bmbouter about 6 years ago

Also note that by having only the orphan cleanup code in the celery task itself, any time the resource manager is waiting for the reservations count to go to 0 will have the task shown as 'waiting'. I think this is good because the user could take an action to cancel the work that is currently running. Also the task runtime of orphan cleanup will be the actual code runtime that will not include the waiting time which is consistent with other tasks.

Actions #13

Updated by jortel@redhat.com about 6 years ago

The changes good. Although, using this special name seems brittle. Perhaps we can add an attribute to the task that stipulates that it needs to be run with zero reservations instead.

The if task.name == "orphan_cleanup":

The orphan clean up task could be long running. I suspect it cannot be canceled, right? What happens if a user tries to cancel the task?

Actions #14

Updated by bmbouter about 6 years ago

  • Description updated (diff)
Actions #15

Updated by jortel@redhat.com about 6 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #16

Updated by jortel@redhat.com about 6 years ago

  • Sprint set to Sprint 34
Actions #17

Updated by daviddavis about 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions #18

Updated by daviddavis about 6 years ago

  • Related to Issue #3508: Artifact files aren't getting deleted along with artifacts added
Actions #20

Updated by daviddavis about 6 years ago

  • Related to Story #3542: As a plugin writer, I can create a task that waits until there are no reservations added
Actions #21

Updated by daviddavis about 6 years ago

Per comment 13, I opened a new issue to handle making this functionality around waiting for no reservations more generic so plugin writers, etc can use it:

https://pulp.plan.io/issues/3542

This should also take care of @jortel's concern about name being being too brittle.

Actions #22

Updated by daviddavis about 6 years ago

  • Status changed from ASSIGNED to POST

Added by daviddavis about 6 years ago

Revision 3eb5b1a2 | View on GitHub

As a user, I can cleanup orphan content and artifacts

fixes #3442 https://pulp.plan.io/issues/3442

Added by daviddavis about 6 years ago

Revision 3eb5b1a2 | View on GitHub

As a user, I can cleanup orphan content and artifacts

fixes #3442 https://pulp.plan.io/issues/3442

Actions #23

Updated by daviddavis about 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #24

Updated by dalley about 6 years ago

  • Related to Story #2484: As a pulp 3.0 user, orphan content delete reports how many units were deleted added
Actions #25

Updated by dkliban@redhat.com about 6 years ago

  • Sprint/Milestone set to 3.0.0
Actions #26

Updated by kersom about 5 years ago

  • Related to Test #4663: Test remove orphans endpoint added
Actions #27

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3, Pulp 3 MVP)
Actions #28

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF