Project

Profile

Help

Task #3923

Remove the GroupDownloader

Added by bmbouter almost 2 years ago. Updated 5 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 42

Description

The GroupDownloader should be deleted.

What it did

The idea with GroupDownloader is that you can create a group of files that belong to content unit and have all the downloads of all the content units happen in parallel.

Why delete it?

asyncio.gather(...) provides the same facility in the standard library: https://docs.python.org/3/library/asyncio-task.html#asyncio.gather

Take your N artifacts for your single content type and wrap them with gather() creating a single co-routine to track them. Then for M content units do the same thing, resulting in M co-routines returned by gather(...). You can schedule these M coroutines which will schedule all downloads conncurrently. Also you can wake up with each of the M coroutines finishing, each representing a completed content unit. This is exactly the same feature set as GroupDownloader.

See it in action

The ArtifactDownloader stage implements the exact alternate pattern described ^ here: https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/stages/artifact_stages.py#L194

Also remove attach_url_to_exception

The attach_url_to_exception decorator only serves the GroupDownloader. It's akward anyway and should also be removed. https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/download/base.py#L29-L55 gather handles exception better than our custom solution did anyway.


Checklist

Associated revisions

Revision d9eb035d View on GitHub
Added by bmbouter over 1 year ago

Remove the GroupDownloader

The GroupDownloader is not needed. See the ticket description for more info.

Also the 'attach_url_to_exception' was only here to serve the GroupDownloader so that is also being removed.

https://pulp.plan.io/issues/3923 closes #3923

Revision d9eb035d View on GitHub
Added by bmbouter over 1 year ago

Remove the GroupDownloader

The GroupDownloader is not needed. See the ticket description for more info.

Also the 'attach_url_to_exception' was only here to serve the GroupDownloader so that is also being removed.

https://pulp.plan.io/issues/3923 closes #3923

History

#1 Updated by dkliban@redhat.com almost 2 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#2 Updated by daviddavis almost 2 years ago

  • Sprint set to Sprint 42

#3 Updated by bmbouter over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

#4 Updated by bmbouter over 1 year ago

  • Status changed from ASSIGNED to POST

#5 Updated by bmbouter over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#6 Updated by daviddavis about 1 year ago

  • Sprint/Milestone set to 3.0.0

#7 Updated by bmbouter about 1 year ago

  • Tags deleted (Pulp 3)

#8 Updated by bmbouter 5 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF