Project

Profile

Help

Story #4174

closed

As a user, I can lazy sync a docker repository

Added by amacdona@redhat.com over 5 years ago. Updated about 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 52
Quarter:

Description

A docker repository is composed from tags, manifests and layers.

We can lazily fetch layers, manifests treated as metadata-like information will be always downloaded to discover the layers.


Related issues

Related to Pulp - Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata filesCLOSED - CURRENTRELEASEmdellweg

Actions
Related to Pulp - Test #4126: Test sync with different download policiesCLOSED - COMPLETEkersomActions
Related to Pulp - Test #4973: Docker - Test sync with different download policiesCLOSED - DUPLICATEActions
Actions #1

Updated by ipanova@redhat.com over 5 years ago

  • Description updated (diff)
Actions #2

Updated by amacdona@redhat.com over 5 years ago

  • Description updated (diff)

Currently, the pulp3 lazy sync implementation plan is to disable the ArtifactDownload and ArtifactSaver stages. This will not work for Docker since these stages are used to download and save metadata (tags, manifests, and manifest lists) as well as blobs/layers (which would not be downloaded or saved for lazy).

We have a couple options to implement this for docker, and it may also be possible to make a contribution to pulpcore to enable easy implementation for other plugins with multiple metadata files.

Idea 1: Add a field to the DeclarativeArtifact

Using a new field, DeclarativeArtifact.metadata (defaults to false), the ArtifactDownload and ArtifactSaver stages could potentially be updated to skip them when da.metadata=false and lazy=true.

Idea 2: Override ArtifactDownload and ArtifactSaver stages

If we override, we just have to add a little logic that checks the type of the DeclarativeContent.content. If lazy and type is Blob, don't download or save.

Actions #3

Updated by amacdona@redhat.com over 5 years ago

  • Related to Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files added
Actions #4

Updated by ipanova@redhat.com almost 5 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 52
Actions #5

Updated by dkliban@redhat.com almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
Actions #6

Updated by dkliban@redhat.com almost 5 years ago

  • Status changed from ASSIGNED to POST

Added by dkliban@redhat.com almost 5 years ago

Revision 75fc7777 | View on GitHub

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Added by dkliban@redhat.com almost 5 years ago

Revision 75fc7777 | View on GitHub

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Added by dkliban@redhat.com almost 5 years ago

Revision 75fc7777 | View on GitHub

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Added by dkliban@redhat.com almost 5 years ago

Revision 75fc7777 | View on GitHub

Problem: docker plugin does not support lazy sync

Solution: add support for lazy sync

This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.

This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.

closes: #4174 https://pulp.plan.io/issues/4174

Actions #7

Updated by dkliban@redhat.com almost 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Applied in changeset commit:pulp_docker|75fc777756b95e1df5da7bd2b08a32c2945742dc.

Actions #8

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3)
Actions #9

Updated by kersom almost 5 years ago

  • Related to Test #4126: Test sync with different download policies added
Actions #10

Updated by kersom almost 5 years ago

  • Related to Test #4973: Docker - Test sync with different download policies added
Actions #11

Updated by ipanova@redhat.com over 4 years ago

  • Project changed from Docker Support to Container Support
Actions #12

Updated by ipanova@redhat.com over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #13

Updated by ipanova@redhat.com about 4 years ago

  • Sprint/Milestone set to 1.0.0

Also available in: Atom PDF