Project

Profile

Help

Story #3497

As a user, I can upload skopeo tar that references existing layers.

Added by tomckay@redhat.com almost 2 years ago. Updated 9 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
2.17.0
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 38

Description

Motivation:

To save bandwidth and improve efficiency, layers referred to in a skopeo tar manifest.json that already exist should be allowed to be omitted from the tar.

As I am implementing "docker push" and "skopeo copy docker:", both tools first check if a layer exists before attempting to push it. Since pulp expects all the layers to be part of the upload this leads to having to get the layer from pulp, add it to the tar file, then give tar file to pulp.

Solution:

The upload API will support uploading archives that contain only a subset of layers needed to create the full container image. Archive must contain manifest.json file. When such an archive is uploaded to Pulp, it will be added to the repository only if the missing layers are already present in the repository. If the layers are missing from both the archive being uploaded and repository the upload will fail.

Associated revisions

Revision 891eff64 View on GitHub
Added by jortel@redhat.com over 1 year ago

Support upload tarball with incomplete images.
closes #3497

Revision 891eff64 View on GitHub
Added by jortel@redhat.com over 1 year ago

Support upload tarball with incomplete images.
closes #3497

Revision 891eff64 View on GitHub
Added by jortel@redhat.com over 1 year ago

Support upload tarball with incomplete images.
closes #3497

Revision 891eff64 View on GitHub
Added by jortel@redhat.com over 1 year ago

Support upload tarball with incomplete images.
closes #3497

History

#1 Updated by dkliban@redhat.com almost 2 years ago

This seems reasonable to me. Our documentation would need to state that this type of upload has a race condition with the orphan cleanup task.

The upload code for pulp_docker would try to find any blobs missing from the traball already in Pulp. If the blob is found, an association with the repository is made. If not found, an error is returned.

#2 Updated by dalley almost 2 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#3 Updated by ipanova@redhat.com almost 2 years ago

Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.

#4 Updated by ipanova@redhat.com over 1 year ago

  • Sprint Candidate changed from No to Yes

#5 Updated by dkliban@redhat.com over 1 year ago

  • Description updated (diff)

#6 Updated by ipanova@redhat.com over 1 year ago

  • Description updated (diff)

#7 Updated by ipanova@redhat.com over 1 year ago

  • Description updated (diff)
  • Groomed changed from No to Yes

#8 Updated by rchan over 1 year ago

  • Sprint set to Sprint 38

#9 Updated by jortel@redhat.com over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#10 Updated by jortel@redhat.com over 1 year ago

wrote:

Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.

@ipanova, here you suggest that a blob not included in the uploaded archive must already exists in pulp AND must already be associated with the destination repository. If the blob exists in pulp but is not associated with the destination repository, why fail the upload instead of associating it as @dkliban suggested in #3497-1? Is your distinction intentional?

#11 Updated by tomckay@redhat.com over 1 year ago

@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.

#12 Updated by jortel@redhat.com over 1 year ago

wrote:

@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.

Okay. Thank you, @ipanova.

#13 Updated by jortel@redhat.com over 1 year ago

  • Status changed from ASSIGNED to POST

#14 Updated by jortel@redhat.com over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#15 Updated by dkliban@redhat.com over 1 year ago

  • Platform Release set to 2.16.2

#16 Updated by dkliban@redhat.com over 1 year ago

  • Platform Release deleted (2.16.2)

#17 Updated by amacdona@redhat.com over 1 year ago

  • Platform Release set to 2.16.2

#18 Updated by amacdona@redhat.com over 1 year ago

  • Platform Release deleted (2.16.2)

#19 Updated by ipanova@redhat.com over 1 year ago

  • Platform Release set to 2.17.0

#20 Updated by rchan over 1 year ago

  • Sprint/Milestone set to 2.17.0

#21 Updated by rchan over 1 year ago

Adding to 2.17.0 milestone. This is one of the required deliverables.

#22 Updated by ipanova@redhat.com over 1 year ago

  • Platform Release changed from 2.17.0 to master

#23 Updated by ipanova@redhat.com over 1 year ago

  • Platform Release deleted (master)

#24 Updated by ipanova@redhat.com over 1 year ago

  • Platform Release set to 2.17.0

#25 Updated by ipanova@redhat.com over 1 year ago

  • Status changed from MODIFIED to ON_QA

#26 Updated by ipanova@redhat.com over 1 year ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#27 Updated by bmbouter 9 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF