Project

Profile

Help

Story #3497

closed

As a user, I can upload skopeo tar that references existing layers.

Added by tomckay@redhat.com over 6 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.17.0
Target Release - Docker:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 38
Quarter:

Description

Motivation:

To save bandwidth and improve efficiency, layers referred to in a skopeo tar manifest.json that already exist should be allowed to be omitted from the tar.

As I am implementing "docker push" and "skopeo copy docker:", both tools first check if a layer exists before attempting to push it. Since pulp expects all the layers to be part of the upload this leads to having to get the layer from pulp, add it to the tar file, then give tar file to pulp.

Solution:

The upload API will support uploading archives that contain only a subset of layers needed to create the full container image. Archive must contain manifest.json file. When such an archive is uploaded to Pulp, it will be added to the repository only if the missing layers are already present in the repository. If the layers are missing from both the archive being uploaded and repository the upload will fail.

Actions #1

Updated by dkliban@redhat.com over 6 years ago

This seems reasonable to me. Our documentation would need to state that this type of upload has a race condition with the orphan cleanup task.

The upload code for pulp_docker would try to find any blobs missing from the traball already in Pulp. If the blob is found, an association with the repository is made. If not found, an error is returned.

Actions #2

Updated by dalley over 6 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0
Actions #3

Updated by ipanova@redhat.com over 6 years ago

Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.

Actions #4

Updated by ipanova@redhat.com over 6 years ago

  • Sprint Candidate changed from No to Yes
Actions #5

Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)
Actions #6

Updated by ipanova@redhat.com over 6 years ago

  • Description updated (diff)
Actions #7

Updated by ipanova@redhat.com over 6 years ago

  • Description updated (diff)
  • Groomed changed from No to Yes
Actions #8

Updated by rchan over 6 years ago

  • Sprint set to Sprint 38
Actions #9

Updated by jortel@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
Actions #10

Updated by jortel@redhat.com over 6 years ago

wrote:

Update: we'll make a query to the destination repo of the upload and check if the blob mentioned in manifest.json is present.
In case it is present in the repo, proceed to the next blob. If the blob is not present, check its existence in the tar file.In case it is not in the tar file - fail upload.

@ipanova, here you suggest that a blob not included in the uploaded archive must already exists in pulp AND must already be associated with the destination repository. If the blob exists in pulp but is not associated with the destination repository, why fail the upload instead of associating it as @dkliban suggested in #3497-1? Is your distinction intentional?

Actions #11

Updated by tomckay@redhat.com over 6 years ago

@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.

Actions #12

Updated by jortel@redhat.com over 6 years ago

wrote:

@jortel - It is intentional. Using this paradigm, a user can't upload a manifest that references layers that they don't really have access to. By forcing the layer to exist in the repository already, this forces the initial upload of the layer as proof of ownership.

Okay. Thank you, @ipanova.

Actions #13

Updated by jortel@redhat.com over 6 years ago

  • Status changed from ASSIGNED to POST

Added by jortel@redhat.com over 6 years ago

Revision 891eff64 | View on GitHub

Support upload tarball with incomplete images. closes #3497

Added by jortel@redhat.com over 6 years ago

Revision 891eff64 | View on GitHub

Support upload tarball with incomplete images. closes #3497

Added by jortel@redhat.com over 6 years ago

Revision 891eff64 | View on GitHub

Support upload tarball with incomplete images. closes #3497

Added by jortel@redhat.com over 6 years ago

Revision 891eff64 | View on GitHub

Support upload tarball with incomplete images. closes #3497

Actions #14

Updated by jortel@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #15

Updated by dkliban@redhat.com over 6 years ago

  • Platform Release set to 2.16.2
Actions #16

Updated by dkliban@redhat.com over 6 years ago

  • Platform Release deleted (2.16.2)
Actions #17

Updated by amacdona@redhat.com over 6 years ago

  • Platform Release set to 2.16.2
Actions #18

Updated by amacdona@redhat.com over 6 years ago

  • Platform Release deleted (2.16.2)
Actions #19

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release set to 2.17.0
Actions #20

Updated by rchan over 6 years ago

  • Sprint/Milestone set to 2.17.0
Actions #21

Updated by rchan over 6 years ago

Adding to 2.17.0 milestone. This is one of the required deliverables.

Actions #22

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release changed from 2.17.0 to master
Actions #23

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release deleted (master)
Actions #24

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release set to 2.17.0
Actions #25

Updated by ipanova@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5
Actions #26

Updated by ipanova@redhat.com about 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #27

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions #28

Updated by ttereshc almost 3 years ago

  • Sprint/Milestone deleted (2.17.0)

Also available in: Atom PDF