Project

Profile

Help

Story #4209

Story #3693: Lazy for Pulp3

As a plugin writer, I can implement lazy sync for plugins with multiple metadata files

Added by amacdona@redhat.com 11 months ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Problem

For plugins such as Python and Docker that have many metadata files, we may need a different way to implement lazy sync than the current plan, which is to omit the ArtifactDownloader stage and the ArtifactSaver stage.

Because there are many metadata files, (and with Docker, these metadata files are also content units), we are using the ArtifactDownloader stage to download metadata. In this case, the metadata files need to be downloaded, and related content saved, but the non-metadata files should be skipped for downloading and saving.

Solution

The DeclarativeArtifact gets a new flag called 'deferred_download' defaulting to False to specify, that the artifact stages stop processing it (making it lazy).

At the same time the 'download_artifacts' flag of DeclarativeVersion is removed.

So the plugin must either use the deferred_download flag, or tailor its own Pipeline.


Related issues

Related to Python Support - Story #1884: As a user, I can lazily sync python packages MODIFIED Actions
Related to Docker Support - Story #4174: As a user, I can lazy sync a docker repository MODIFIED Actions
Related to RPM Support - Issue #4446: multiple test failures on master branch MODIFIED Actions

Associated revisions

Revision a6913878 View on GitHub
Added by mdellweg 8 months ago

Add deferred_download flag to DeclarativeArtifact

Required PR: https://github.com/pulp/pulp_file/pull/177

https://pulp.plan.io/issues/4209
re #4209

Revision 354aa520 View on GitHub
Added by bmbouter 8 months ago

Updating lazy for pulp_rpm

The changes to the plugin API have changed the DeclarativeVersion
interface. We now set deferred_download on the DeclarativeArtifact
directly.

https://pulp.plan.io/issues/4209
re #4209

History

#1 Updated by dalley 11 months ago

  • Sprint set to Sprint 46
  • Tags Pulp 3 RC Blocker added

I'm pretty sure we meant to make this an RC blocker but forgot to do so. Adding to the sprint because that's what we said we would do with all RC blockers IIRC.

#2 Updated by dalley 11 months ago

  • Related to Story #1884: As a user, I can lazily sync python packages added

#3 Updated by amacdona@redhat.com 11 months ago

  • Related to Story #4174: As a user, I can lazy sync a docker repository added

#4 Updated by bmbouter 11 months ago

  • Sprint deleted (Sprint 46)

I have to comment on this as part of the lazy epic, but since there are two options and it's not clear which we are doing I'm going to remove from the sprint for right now. At the RC blocker meeting we wanted work on the sprint to be ready-to-work. Once some more comments happen we will put it back on the sprint (very soon).

#5 Updated by bmbouter 10 months ago

I agree with the problem this story identifies. We need a way for plugin to add lazy support while still using the ArtifactDownloader and ArtifactSaver stages in the pipeline.

There is a related problem that we need to consider at the same times as this one because I believe one solution will resolve all of these issues. Plugins that discover content to download from content downloaded by the pipeline would need additional ArtifactDownloader and ArtifactSaver stages later in the pipeline. Adding more stages later fixes that issue, but since the pipeline size has to be fixed and the number of "discoveries" does not this solution can't address all plugin needs. For example the Maven sync will have this issue.

I'm writing ^ issue up as a separate ticket now and after that I can show how that solution will also resolve this problem as well.

#6 Updated by jortel@redhat.com 10 months ago

Option 1 seems simplest only let's name it something like DeclarativeArtifact.always_download instead. This way the attribute describes the desired behavior instead of describing the artifact which has the side effect of the desired behavior.

#7 Updated by mdellweg 10 months ago

Just to be explicit here: Please ensure, that this is handled per Artifact. In gem, a content unit consists of two artifacts: One containing the actual gem (subject to lazy) and one containing Metadata (which in a future version might be desirable to be available immediately for dependency resolution).

BTW: Option 1 +1

#8 Updated by mdellweg 10 months ago

If you decide for Option 1, i have a solution ready:
https://github.com/ATIX-AG/pulpcore-plugin/tree/non_lazy_artifacts

#9 Updated by bmbouter 8 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mdellweg

Even though the full design isn't written out here, it is being discussed here: https://github.com/pulp/pulpcore-plugin/pull/50

So others don't start on the work I'm marking it as assigned now.

#10 Updated by mdellweg 8 months ago

The solution agreed upon now is:

The DeclarativeArtifact gets a new flag called 'deferred_download' defaulting to False to specify, that the artifact stages stop processing it (making it lazy).
At the same time the 'download_artifacts' flag of DeclarativeVersion is removed.
So the plugin must either use the deferred_download flag, or tailor its own Pipeline.

#11 Updated by bmbouter 8 months ago

  • Description updated (diff)

#12 Updated by bmbouter 8 months ago

  • Status changed from ASSIGNED to POST

#13 Updated by bmbouter 8 months ago

  • Status changed from POST to MODIFIED

@mdellweg can you closes #4209 with the docs PR you are making please?

#14 Updated by ttereshc 8 months ago

  • Related to Issue #4446: multiple test failures on master branch added

#15 Updated by mdellweg 8 months ago

@bmbouter Adding the closes tag brings travis to fail with:

Error: issue #4209 has invalid status of MODIFIED. Status must be one of NEW, ASSIGNED, POST.

Removed it again.

#16 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#17 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3, Pulp 3 RC Blocker)

Please register to edit this issue

Also available in: Atom PDF