Story #3497
closedAs a user, I can upload skopeo tar that references existing layers.
100%
Description
Motivation:
To save bandwidth and improve efficiency, layers referred to in a skopeo tar manifest.json that already exist should be allowed to be omitted from the tar.
As I am implementing "docker push" and "skopeo copy docker:", both tools first check if a layer exists before attempting to push it. Since pulp expects all the layers to be part of the upload this leads to having to get the layer from pulp, add it to the tar file, then give tar file to pulp.
Solution:
The upload API will support uploading archives that contain only a subset of layers needed to create the full container image. Archive must contain manifest.json file. When such an archive is uploaded to Pulp, it will be added to the repository only if the missing layers are already present in the repository. If the layers are missing from both the archive being uploaded and repository the upload will fail.
Updated by dkliban@redhat.com over 6 years ago
This seems reasonable to me. Our documentation would need to state that this type of upload has a race condition with the orphan cleanup task.
The upload code for pulp_docker would try to find any blobs missing from the traball already in Pulp. If the blob is found, an association with the repository is made. If not found, an error is returned.
Updated by dalley over 6 years ago
- Tracker changed from Issue to Story
- % Done set to 0
Updated by ipanova@redhat.com over 6 years ago
Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.
Updated by ipanova@redhat.com over 6 years ago
- Sprint Candidate changed from No to Yes
Updated by ipanova@redhat.com over 6 years ago
- Description updated (diff)
- Groomed changed from No to Yes
Updated by jortel@redhat.com over 6 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to jortel@redhat.com
Updated by jortel@redhat.com over 6 years ago
ipanova@redhat.com wrote:
Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.
@ipanova, here you suggest that a blob not included in the uploaded archive must already exists in pulp AND must already be associated with the destination repository. If the blob exists in pulp but is not associated with the destination repository, why fail the upload instead of associating it as @dkliban suggested in #3497-1? Is your distinction intentional?
Updated by tomckay@redhat.com over 6 years ago
@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.
Updated by jortel@redhat.com over 6 years ago
tomckay@redhat.com wrote:
@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.
Okay. Thank you, @ipanova.
Updated by jortel@redhat.com over 6 years ago
- Status changed from ASSIGNED to POST
Added by jortel@redhat.com over 6 years ago
Added by jortel@redhat.com over 6 years ago
Revision 891eff64 | View on GitHub
Support upload tarball with incomplete images. closes #3497
Added by jortel@redhat.com over 6 years ago
Revision 891eff64 | View on GitHub
Support upload tarball with incomplete images. closes #3497
Added by jortel@redhat.com over 6 years ago
Revision 891eff64 | View on GitHub
Support upload tarball with incomplete images. closes #3497
Updated by jortel@redhat.com over 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset 891eff64b28ef1cc37faf4609635fda615cadb3a.
Updated by dkliban@redhat.com over 6 years ago
- Platform Release deleted (
2.16.2)
Updated by amacdona@redhat.com over 6 years ago
- Platform Release deleted (
2.16.2)
Updated by rchan over 6 years ago
Adding to 2.17.0 milestone. This is one of the required deliverables.
Updated by ipanova@redhat.com over 6 years ago
- Platform Release changed from 2.17.0 to master
Updated by ipanova@redhat.com over 6 years ago
- Platform Release deleted (
master)
Updated by ipanova@redhat.com over 6 years ago
- Status changed from MODIFIED to 5
Updated by ipanova@redhat.com about 6 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Support upload tarball with incomplete images. closes #3497