Project

Profile

Help

Story #4209

Story #3693: Lazy for Pulp3

As a plugin writer, I can implement lazy sync for plugins with multiple metadata files

Added by amacdona@redhat.com over 1 year ago. Updated 3 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Problem

For plugins such as Python and Docker that have many metadata files, we may need a different way to implement lazy sync than the current plan, which is to omit the ArtifactDownloader stage and the ArtifactSaver stage.

Because there are many metadata files, (and with Docker, these metadata files are also content units), we are using the ArtifactDownloader stage to download metadata. In this case, the metadata files need to be downloaded, and related content saved, but the non-metadata files should be skipped for downloading and saving.

Solution

The DeclarativeArtifact gets a new flag called 'deferred_download' defaulting to False to specify, that the artifact stages stop processing it (making it lazy).

At the same time the 'download_artifacts' flag of DeclarativeVersion is removed.

So the plugin must either use the deferred_download flag, or tailor its own Pipeline.


Related issues

Related to Python Support - Story #1884: As a user, I can lazily sync python packages MODIFIED Actions
Related to Container Support - Story #4174: As a user, I can lazy sync a docker repository CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Issue #4446: multiple test failures on master branch CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision 354aa520 View on GitHub
Added by bmbouter about 1 year ago

Updating lazy for pulp_rpm

The changes to the plugin API have changed the DeclarativeVersion interface. We now set deferred_download on the DeclarativeArtifact directly.

https://pulp.plan.io/issues/4209 re #4209

History

#1 Updated by dalley about 1 year ago

  • Sprint set to Sprint 46
  • Tags Pulp 3 RC Blocker added

I'm pretty sure we meant to make this an RC blocker but forgot to do so. Adding to the sprint because that's what we said we would do with all RC blockers IIRC.

#2 Updated by dalley about 1 year ago

  • Related to Story #1884: As a user, I can lazily sync python packages added

#3 Updated by amacdona@redhat.com about 1 year ago

  • Related to Story #4174: As a user, I can lazy sync a docker repository added

#4 Updated by bmbouter about 1 year ago

  • Sprint deleted (Sprint 46)

I have to comment on this as part of the lazy epic, but since there are two options and it's not clear which we are doing I'm going to remove from the sprint for right now. At the RC blocker meeting we wanted work on the sprint to be ready-to-work. Once some more comments happen we will put it back on the sprint (very soon).

#5 Updated by bmbouter about 1 year ago

I agree with the problem this story identifies. We need a way for plugin to add lazy support while still using the ArtifactDownloader and ArtifactSaver stages in the pipeline.

There is a related problem that we need to consider at the same times as this one because I believe one solution will resolve all of these issues. Plugins that discover content to download from content downloaded by the pipeline would need additional ArtifactDownloader and ArtifactSaver stages later in the pipeline. Adding more stages later fixes that issue, but since the pipeline size has to be fixed and the number of "discoveries" does not this solution can't address all plugin needs. For example the Maven sync will have this issue.

I'm writing ^ issue up as a separate ticket now and after that I can show how that solution will also resolve this problem as well.

#6 Updated by jortel@redhat.com about 1 year ago

Option 1 seems simplest only let's name it something like DeclarativeArtifact.always_download instead. This way the attribute describes the desired behavior instead of describing the artifact which has the side effect of the desired behavior.

#7 Updated by mdellweg about 1 year ago

Just to be explicit here: Please ensure, that this is handled per Artifact. In gem, a content unit consists of two artifacts: One containing the actual gem (subject to lazy) and one containing Metadata (which in a future version might be desirable to be available immediately for dependency resolution).

BTW: Option 1 +1

#8 Updated by mdellweg about 1 year ago

If you decide for Option 1, i have a solution ready:
https://github.com/ATIX-AG/pulpcore-plugin/tree/non_lazy_artifacts

#9 Updated by bmbouter about 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mdellweg

Even though the full design isn't written out here, it is being discussed here: https://github.com/pulp/pulpcore-plugin/pull/50

So others don't start on the work I'm marking it as assigned now.

#10 Updated by mdellweg about 1 year ago

The solution agreed upon now is:

The DeclarativeArtifact gets a new flag called 'deferred_download' defaulting to False to specify, that the artifact stages stop processing it (making it lazy).
At the same time the 'download_artifacts' flag of DeclarativeVersion is removed.
So the plugin must either use the deferred_download flag, or tailor its own Pipeline.

#11 Updated by bmbouter about 1 year ago

  • Description updated (diff)

#12 Updated by bmbouter about 1 year ago

  • Status changed from ASSIGNED to POST

#13 Updated by bmbouter about 1 year ago

  • Status changed from POST to MODIFIED

@mdellweg can you closes #4209 with the docs PR you are making please?

#14 Updated by ttereshc about 1 year ago

  • Related to Issue #4446: multiple test failures on master branch added

#15 Updated by mdellweg 12 months ago

@bmbouter Adding the closes tag brings travis to fail with:

Error: issue #4209 has invalid status of MODIFIED. Status must be one of NEW, ASSIGNED, POST.

Removed it again.

#16 Updated by daviddavis 10 months ago

  • Sprint/Milestone set to 3.0.0

#17 Updated by bmbouter 10 months ago

  • Tags deleted (Pulp 3, Pulp 3 RC Blocker)

#18 Updated by bmbouter 3 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF