Story #4174
closed
As a user, I can lazy sync a docker repository
100%
Description
A docker repository is composed from tags, manifests and layers.
We can lazily fetch layers, manifests treated as metadata-like information will be always downloaded to discover the layers.
Related issues
Updated by amacdona@redhat.com about 6 years ago
- Description updated (diff)
Currently, the pulp3 lazy sync implementation plan is to disable the ArtifactDownload and ArtifactSaver stages. This will not work for Docker since these stages are used to download and save metadata (tags, manifests, and manifest lists) as well as blobs/layers (which would not be downloaded or saved for lazy).
We have a couple options to implement this for docker, and it may also be possible to make a contribution to pulpcore to enable easy implementation for other plugins with multiple metadata files.
Idea 1: Add a field to the DeclarativeArtifact
Using a new field, DeclarativeArtifact.metadata (defaults to false), the ArtifactDownload and ArtifactSaver stages could potentially be updated to skip them when da.metadata=false and lazy=true.
Idea 2: Override ArtifactDownload and ArtifactSaver stages
If we override, we just have to add a little logic that checks the type of the DeclarativeContent.content. If lazy and type is Blob, don't download or save.
Updated by amacdona@redhat.com about 6 years ago
- Related to Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata files added
Updated by ipanova@redhat.com over 5 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
- Sprint set to Sprint 52
Updated by dkliban@redhat.com over 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
Updated by dkliban@redhat.com over 5 years ago
- Status changed from ASSIGNED to POST
Added by dkliban@redhat.com over 5 years ago
Added by dkliban@redhat.com over 5 years ago
Revision 75fc7777 | View on GitHub
Problem: docker plugin does not support lazy sync
Solution: add support for lazy sync
This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.
This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.
Added by dkliban@redhat.com over 5 years ago
Revision 75fc7777 | View on GitHub
Problem: docker plugin does not support lazy sync
Solution: add support for lazy sync
This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.
This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.
Added by dkliban@redhat.com over 5 years ago
Revision 75fc7777 | View on GitHub
Problem: docker plugin does not support lazy sync
Solution: add support for lazy sync
This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.
This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.
Updated by dkliban@redhat.com over 5 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset commit:pulp_docker|75fc777756b95e1df5da7bd2b08a32c2945742dc.
Updated by kersom over 5 years ago
- Related to Test #4126: Test sync with different download policies added
Updated by kersom over 5 years ago
- Related to Test #4973: Docker - Test sync with different download policies added
Updated by ipanova@redhat.com about 5 years ago
- Project changed from Docker Support to Container Support
Updated by ipanova@redhat.com about 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Problem: docker plugin does not support lazy sync
Solution: add support for lazy sync
This patch adds ability to create remotes with policy='on_demand'. When such a remote is used to sync a repository, the Blobs are not downloaded. Only manifests and manifest lists are saved at sync time. Blobs are retrieved on demand from clients.
This patch also includes a set of tests that check that you can pull content from a repo that was synced using a remote that has policy set to 'on_demand'. The tests assert that performing a docker/podman pull results in more artifacts being stored in Pulp than before the image was requested.
closes: #4174 https://pulp.plan.io/issues/4174