Project

Profile

Help

Story #3497

As a user, I can upload skopeo tar that references existing layers.

Added by tomckay@redhat.com about 1 year ago. Updated 5 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
% Done:

100%

Platform Release:
2.17.0
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 38

Description

Motivation:

To save bandwidth and improve efficiency, layers referred to in a skopeo tar manifest.json that already exist should be allowed to be omitted from the tar.

As I am implementing "docker push" and "skopeo copy docker:", both tools first check if a layer exists before attempting to push it. Since pulp expects all the layers to be part of the upload this leads to having to get the layer from pulp, add it to the tar file, then give tar file to pulp.

Solution:

The upload API will support uploading archives that contain only a subset of layers needed to create the full container image. Archive must contain manifest.json file. When such an archive is uploaded to Pulp, it will be added to the repository only if the missing layers are already present in the repository. If the layers are missing from both the archive being uploaded and repository the upload will fail.

Associated revisions

Revision 891eff64 View on GitHub
Added by jortel@redhat.com 11 months ago

Support upload tarball with incomplete images.
closes #3497

Revision 891eff64 View on GitHub
Added by jortel@redhat.com 11 months ago

Support upload tarball with incomplete images.
closes #3497

History

#1 Updated by dkliban@redhat.com about 1 year ago

This seems reasonable to me. Our documentation would need to state that this type of upload has a race condition with the orphan cleanup task.

The upload code for pulp_docker would try to find any blobs missing from the traball already in Pulp. If the blob is found, an association with the repository is made. If not found, an error is returned.

#2 Updated by dalley about 1 year ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#3 Updated by ipanova@redhat.com about 1 year ago

Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.

#4 Updated by ipanova@redhat.com 11 months ago

  • Sprint Candidate changed from No to Yes

#5 Updated by dkliban@redhat.com 11 months ago

  • Description updated (diff)

#6 Updated by ipanova@redhat.com 11 months ago

  • Description updated (diff)

#7 Updated by ipanova@redhat.com 11 months ago

  • Description updated (diff)
  • Groomed changed from No to Yes

#8 Updated by rchan 11 months ago

  • Sprint set to Sprint 38

#9 Updated by jortel@redhat.com 11 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#10 Updated by jortel@redhat.com 11 months ago

wrote:

Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.

@ipanova, here you suggest that a blob not included in the uploaded archive must already exists in pulp AND must already be associated with the destination repository. If the blob exists in pulp but is not associated with the destination repository, why fail the upload instead of associating it as @dkliban suggested in #3497-1? Is your distinction intentional?

#11 Updated by tomckay@redhat.com 11 months ago

@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.

#12 Updated by jortel@redhat.com 11 months ago

wrote:

@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.

Okay. Thank you, @ipanova.

#13 Updated by jortel@redhat.com 11 months ago

  • Status changed from ASSIGNED to POST

#14 Updated by jortel@redhat.com 10 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#15 Updated by dkliban@redhat.com 10 months ago

  • Platform Release set to 2.16.2

#16 Updated by dkliban@redhat.com 10 months ago

  • Platform Release deleted (2.16.2)

#17 Updated by amacdona@redhat.com 10 months ago

  • Platform Release set to 2.16.2

#18 Updated by amacdona@redhat.com 10 months ago

  • Platform Release deleted (2.16.2)

#19 Updated by ipanova@redhat.com 10 months ago

  • Platform Release set to 2.17.0

#20 Updated by rchan 10 months ago

  • Sprint/Milestone set to 2.17.0

#21 Updated by rchan 10 months ago

Adding to 2.17.0 milestone. This is one of the required deliverables.

#22 Updated by ipanova@redhat.com 9 months ago

  • Platform Release changed from 2.17.0 to master

#23 Updated by ipanova@redhat.com 9 months ago

  • Platform Release deleted (master)

#24 Updated by ipanova@redhat.com 9 months ago

  • Platform Release set to 2.17.0

#25 Updated by ipanova@redhat.com 8 months ago

  • Status changed from MODIFIED to ON_QA

#26 Updated by ipanova@redhat.com 8 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#27 Updated by bmbouter 5 days ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF