Project

Profile

Help

Story #4174

As a user, I can lazy sync a docker repository

Added by amacdona@redhat.com about 1 year ago. Updated 7 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 52

Description

A docker repository is composed from tags, manifests and layers.

We can lazily fetch layers, manifests treated as metadata-like information will be always downloaded to discover the layers.


Related issues

Related to Pulp - Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files CLOSED - CURRENTRELEASE Actions
Related to Pulp - Test #4126: Test sync with different download policies CLOSED - COMPLETE Actions
Related to Pulp - Test #4973: Docker - Test sync with different download policies NEW Actions

Associated revisions

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com 9 months ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to
sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at
sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that
was synced using a remote that has policy set to 'on_demand'. The tests assert that performing
a docker/podman pull results in more artifacts being stored in Pulp than before the image was
requested.

closes: #4174
https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com 9 months ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to
sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at
sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that
was synced using a remote that has policy set to 'on_demand'. The tests assert that performing
a docker/podman pull results in more artifacts being stored in Pulp than before the image was
requested.

closes: #4174
https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com 9 months ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to
sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at
sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that
was synced using a remote that has policy set to 'on_demand'. The tests assert that performing
a docker/podman pull results in more artifacts being stored in Pulp than before the image was
requested.

closes: #4174
https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com 9 months ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to
sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at
sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that
was synced using a remote that has policy set to 'on_demand'. The tests assert that performing
a docker/podman pull results in more artifacts being stored in Pulp than before the image was
requested.

closes: #4174
https://pulp.plan.io/issues/4174

History

#1 Updated by ipanova@redhat.com about 1 year ago

  • Description updated (diff)

#2 Updated by amacdona@redhat.com about 1 year ago

  • Description updated (diff)

Currently, the pulp3 lazy sync implementation plan is to disable the ArtifactDownload and ArtifactSaver stages. This will not work for Docker since these stages are used to download and save metadata (tags, manifests, and manifest lists) as well as blobs/layers (which would not be downloaded or saved for lazy).

We have a couple options to implement this for docker, and it may also be possible to make a contribution to pulpcore to enable easy implementation for other plugins with multiple metadata files.

Idea 1: Add a field to the DeclarativeArtifact

Using a new field, DeclarativeArtifact.metadata (defaults to false), the ArtifactDownload and ArtifactSaver stages could potentially be updated to skip them when da.metadata=false and lazy=true.

Idea 2: Override ArtifactDownload and ArtifactSaver stages

If we override, we just have to add a little logic that checks the type of the DeclarativeContent.content. If lazy and type is Blob, don't download or save.

#3 Updated by amacdona@redhat.com about 1 year ago

  • Related to Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files added

#4 Updated by ipanova@redhat.com 10 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 52

#5 Updated by dkliban@redhat.com 10 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#6 Updated by dkliban@redhat.com 10 months ago

  • Status changed from ASSIGNED to POST

#7 Updated by dkliban@redhat.com 9 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Applied in changeset commit:pulp_docker|75fc777756b95e1df5da7bd2b08a32c2945742dc.

#8 Updated by bmbouter 9 months ago

  • Tags deleted (Pulp 3)

#9 Updated by kersom 9 months ago

  • Related to Test #4126: Test sync with different download policies added

#10 Updated by kersom 8 months ago

  • Related to Test #4973: Docker - Test sync with different download policies added

#11 Updated by ipanova@redhat.com 2 months ago

  • Project changed from Docker Support to Container Support

#12 Updated by ipanova@redhat.com about 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#13 Updated by ipanova@redhat.com 7 days ago

  • Sprint/Milestone set to 1.0.0

Please register to edit this issue

Also available in: Atom PDF