Project

Profile

Help

Story #4174

As a user, I can lazy sync a docker repository

Added by amacdona@redhat.com almost 2 years ago. Updated 8 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 52
Quarter:

Description

A docker repository is composed from tags, manifests and layers.

We can lazily fetch layers, manifests treated as metadata-like information will be always downloaded to discover the layers.


Related issues

Related to Pulp - Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata filesCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Test #4126: Test sync with different download policiesCLOSED - COMPLETE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Test #4973: Docker - Test sync with different download policiesNEW<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com over 1 year ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com over 1 year ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com over 1 year ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Revision 75fc7777 View on GitHub
Added by dkliban@redhat.com over 1 year ago

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

History

#1 Updated by ipanova@redhat.com almost 2 years ago

  • Description updated (diff)

#2 Updated by amacdona@redhat.com almost 2 years ago

  • Description updated (diff)

Currently, the pulp3 lazy sync implementation plan is to disable the ArtifactDownload and ArtifactSaver stages. This will not work for Docker since these stages are used to download and save metadata (tags, manifests, and manifest lists) as well as blobs/layers (which would not be downloaded or saved for lazy).

We have a couple options to implement this for docker, and it may also be possible to make a contribution to pulpcore to enable easy implementation for other plugins with multiple metadata files.

Idea 1: Add a field to the DeclarativeArtifact

Using a new field, DeclarativeArtifact.metadata (defaults to false), the ArtifactDownload and ArtifactSaver stages could potentially be updated to skip them when da.metadata=false and lazy=true.

Idea 2: Override ArtifactDownload and ArtifactSaver stages

If we override, we just have to add a little logic that checks the type of the DeclarativeContent.content. If lazy and type is Blob, don't download or save.

#3 Updated by amacdona@redhat.com almost 2 years ago

  • Related to Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files added

#4 Updated by ipanova@redhat.com over 1 year ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 52

#5 Updated by dkliban@redhat.com over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#6 Updated by dkliban@redhat.com over 1 year ago

  • Status changed from ASSIGNED to POST

#7 Updated by dkliban@redhat.com over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Applied in changeset commit:pulp_docker|75fc777756b95e1df5da7bd2b08a32c2945742dc.

#8 Updated by bmbouter over 1 year ago

  • Tags deleted (Pulp 3)

#9 Updated by kersom over 1 year ago

  • Related to Test #4126: Test sync with different download policies added

#10 Updated by kersom over 1 year ago

  • Related to Test #4973: Docker - Test sync with different download policies added

#11 Updated by ipanova@redhat.com 10 months ago

  • Project changed from Docker Support to Container Support

#12 Updated by ipanova@redhat.com 10 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#13 Updated by ipanova@redhat.com 8 months ago

  • Sprint/Milestone set to 1.0.0

Please register to edit this issue

Also available in: Atom PDF