Story #4209
closed
Story #3693: Lazy for Pulp3
As a plugin writer, I can implement lazy sync for plugins with multiple metadata files
0%
Description
Problem¶
For plugins such as Python and Docker that have many metadata files, we may need a different way to implement lazy sync than the current plan, which is to omit the ArtifactDownloader stage and the ArtifactSaver stage.
Because there are many metadata files, (and with Docker, these metadata files are also content units), we are using the ArtifactDownloader stage to download metadata. In this case, the metadata files need to be downloaded, and related content saved, but the non-metadata files should be skipped for downloading and saving.
Solution¶
The DeclarativeArtifact gets a new flag called 'deferred_download' defaulting to False to specify, that the artifact stages stop processing it (making it lazy).
At the same time the 'download_artifacts' flag of DeclarativeVersion is removed.
So the plugin must either use the deferred_download flag, or tailor its own Pipeline.
Related issues
Updated by dalley over 5 years ago
- Sprint set to Sprint 46
- Tags Pulp 3 RC Blocker added
I'm pretty sure we meant to make this an RC blocker but forgot to do so. Adding to the sprint because that's what we said we would do with all RC blockers IIRC.
Updated by dalley over 5 years ago
- Related to Story #1884: As a user, I can lazily sync python packages added
Updated by amacdona@redhat.com over 5 years ago
- Related to Story #4174: As a user, I can lazy sync a docker repository added
Updated by bmbouter over 5 years ago
- Sprint deleted (
Sprint 46)
I have to comment on this as part of the lazy epic, but since there are two options and it's not clear which we are doing I'm going to remove from the sprint for right now. At the RC blocker meeting we wanted work on the sprint to be ready-to-work. Once some more comments happen we will put it back on the sprint (very soon).
Updated by bmbouter over 5 years ago
I agree with the problem this story identifies. We need a way for plugin to add lazy support while still using the ArtifactDownloader and ArtifactSaver stages in the pipeline.
There is a related problem that we need to consider at the same times as this one because I believe one solution will resolve all of these issues. Plugins that discover content to download from content downloaded by the pipeline would need additional ArtifactDownloader and ArtifactSaver stages later in the pipeline. Adding more stages later fixes that issue, but since the pipeline size has to be fixed and the number of "discoveries" does not this solution can't address all plugin needs. For example the Maven sync will have this issue.
I'm writing ^ issue up as a separate ticket now and after that I can show how that solution will also resolve this problem as well.
Updated by jortel@redhat.com over 5 years ago
Option 1 seems simplest only let's name it something like DeclarativeArtifact.always_download instead. This way the attribute describes the desired behavior instead of describing the artifact which has the side effect of the desired behavior.
Updated by mdellweg over 5 years ago
Just to be explicit here: Please ensure, that this is handled per Artifact. In gem, a content unit consists of two artifacts: One containing the actual gem (subject to lazy) and one containing Metadata (which in a future version might be desirable to be available immediately for dependency resolution).
BTW: Option 1 +1
Updated by mdellweg over 5 years ago
If you decide for Option 1, i have a solution ready:
https://github.com/ATIX-AG/pulpcore-plugin/tree/non_lazy_artifacts
Updated by bmbouter about 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to mdellweg
Even though the full design isn't written out here, it is being discussed here: https://github.com/pulp/pulpcore-plugin/pull/50
So others don't start on the work I'm marking it as assigned now.
Added by mdellweg about 5 years ago
Updated by mdellweg about 5 years ago
The solution agreed upon now is:¶
The DeclarativeArtifact gets a new flag called 'deferred_download' defaulting to False to specify, that the artifact stages stop processing it (making it lazy).
At the same time the 'download_artifacts' flag of DeclarativeVersion is removed.
So the plugin must either use the deferred_download flag, or tailor its own Pipeline.
Added by bmbouter about 5 years ago
Revision 354aa520 | View on GitHub
Updating lazy for pulp_rpm
The changes to the plugin API have changed the DeclarativeVersion interface. We now set deferred_download on the DeclarativeArtifact directly.
Updated by bmbouter about 5 years ago
- Status changed from ASSIGNED to POST
Updated by bmbouter about 5 years ago
- Status changed from POST to MODIFIED
mdellweg can you closes #4209 with the docs PR you are making please?
Updated by ttereshc about 5 years ago
- Related to Issue #4446: multiple test failures on master branch added
Updated by mdellweg about 5 years ago
bmbouter Adding the closes tag brings travis to fail with:
Error: issue #4209 has invalid status of MODIFIED. Status must be one of NEW, ASSIGNED, POST.
Removed it again.
Updated by bmbouter almost 5 years ago
- Tags deleted (
Pulp 3, Pulp 3 RC Blocker)
Updated by bmbouter over 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Use deferred_download flag for lazy feature
Required PR: https://github.com/pulp/pulpcore-plugin/pull/50
https://pulp.plan.io/issues/4209 re #4209