Story #3497
closed
As a user, I can upload skopeo tar that references existing layers.
Status:
CLOSED - CURRENTRELEASE
Description
Motivation:
To save bandwidth and improve efficiency, layers referred to in a skopeo tar manifest.json that already exist should be allowed to be omitted from the tar.
As I am implementing "docker push" and "skopeo copy docker:", both tools first check if a layer exists before attempting to push it. Since pulp expects all the layers to be part of the upload this leads to having to get the layer from pulp, add it to the tar file, then give tar file to pulp.
Solution:
The upload API will support uploading archives that contain only a subset of layers needed to create the full container image. Archive must contain manifest.json file. When such an archive is uploaded to Pulp, it will be added to the repository only if the missing layers are already present in the repository. If the layers are missing from both the archive being uploaded and repository the upload will fail.
This seems reasonable to me. Our documentation would need to state that this type of upload has a race condition with the orphan cleanup task.
The upload code for pulp_docker would try to find any blobs missing from the traball already in Pulp. If the blob is found, an association with the repository is made. If not found, an error is returned.
- Tracker changed from Issue to Story
- % Done set to 0
Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.
- Sprint Candidate changed from No to Yes
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
- Groomed changed from No to Yes
- Status changed from NEW to ASSIGNED
- Assignee set to jortel@redhat.com
ipanova@redhat.com wrote:
Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.
@ipanova, here you suggest that a blob not included in the uploaded archive must already exists in pulp AND must already be associated with the destination repository. If the blob exists in pulp but is not associated with the destination repository, why fail the upload instead of associating it as @dkliban suggested in #3497-1? Is your distinction intentional?
@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.
tomckay@redhat.com wrote:
@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.
Okay. Thank you, @ipanova.
- Status changed from ASSIGNED to POST
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Platform Release set to 2.16.2
- Platform Release deleted (
2.16.2)
- Platform Release set to 2.16.2
- Platform Release deleted (
2.16.2)
- Platform Release set to 2.17.0
- Sprint/Milestone set to 2.17.0
Adding to 2.17.0 milestone. This is one of the required deliverables.
- Platform Release changed from 2.17.0 to master
- Platform Release deleted (
master)
- Platform Release set to 2.17.0
- Status changed from MODIFIED to 5
- Status changed from 5 to CLOSED - CURRENTRELEASE
- Sprint/Milestone deleted (
2.17.0)
Also available in: Atom
PDF
.
Support upload tarball with incomplete images. closes #3497