Project

Profile

Help

Story #4174

As a user, I can lazy sync a docker repository

Added by amacdona@redhat.com 12 months ago. Updated 7 months ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 52

Description

A docker repository is composed from tags, manifests and layers.

We can lazily fetch layers, manifests treated as metadata-like information will be always downloaded to discover the layers.


Related issues

Related to Pulp - Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files MODIFIED Actions
Related to Pulp - Test #4126: Test sync with different download policies CLOSED - COMPLETE Actions
Related to Pulp - Test #4973: Docker - Test sync with different download policies NEW Actions

Associated revisions

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com 7 months ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to
sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at
sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that
was synced using a remote that has policy set to 'on_demand'. The tests assert that performing
a docker/podman pull results in more artifacts being stored in Pulp than before the image was
requested.

closes: #4174
https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com 7 months ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to
sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at
sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that
was synced using a remote that has policy set to 'on_demand'. The tests assert that performing
a docker/podman pull results in more artifacts being stored in Pulp than before the image was
requested.

closes: #4174
https://pulp.plan.io/issues/4174

History

#1 Updated by ipanova@redhat.com 12 months ago

  • Description updated (diff)

#2 Updated by amacdona@redhat.com 12 months ago

  • Description updated (diff)

Currently, the pulp3 lazy sync implementation plan is to disable the ArtifactDownload and ArtifactSaver stages. This will not work for Docker since these stages are used to download and save metadata (tags, manifests, and manifest lists) as well as blobs/layers (which would not be downloaded or saved for lazy).

We have a couple options to implement this for docker, and it may also be possible to make a contribution to pulpcore to enable easy implementation for other plugins with multiple metadata files.

Idea 1: Add a field to the DeclarativeArtifact

Using a new field, DeclarativeArtifact.metadata (defaults to false), the ArtifactDownload and ArtifactSaver stages could potentially be updated to skip them when da.metadata=false and lazy=true.

Idea 2: Override ArtifactDownload and ArtifactSaver stages

If we override, we just have to add a little logic that checks the type of the DeclarativeContent.content. If lazy and type is Blob, don't download or save.

#3 Updated by amacdona@redhat.com 11 months ago

  • Related to Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files added

#4 Updated by ipanova@redhat.com 7 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 52

#5 Updated by dkliban@redhat.com 7 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#6 Updated by dkliban@redhat.com 7 months ago

  • Status changed from ASSIGNED to POST

#7 Updated by dkliban@redhat.com 7 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

#9 Updated by kersom 7 months ago

  • Related to Test #4126: Test sync with different download policies added

#10 Updated by kersom 5 months ago

  • Related to Test #4973: Docker - Test sync with different download policies added

Please register to edit this issue

Also available in: Atom PDF