Project

Profile

Help

Story #5027

closed

As a user, I can 'docker push' to a Pulp repository

Added by dkliban@redhat.com almost 5 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Motivation

There are currently two upload workflows for Pulp 3 users. Most plugins are encouraged to provide support for the latter.

Basic Upload

1. Create a repository named 'foo'
2. Upload an artifact
3. Create content unit
4. Create a new repository version of 'foo' from the content unit

Smart Upload

1. Create a repository name 'foo'
2. Upload content unit and create a repository version

The Smart Upload for the Docker plugin needs to support creating repository versions using a docker client.

Smart Upload for Docker

1. Create a repository named 'foo'
2. docker tag docker tag 0e5574283393 <pulp-hostname>:24817/foo
3. docker push <pulp-hostname>:24817/foo

Step 3 above creates a new repository version of 'foo' that contains a manifest '0e5574283393', a tag 'latest' pointing to the manifest, and all other manifests and blobs associated with manifest '0e5574283393'. The latest repository version of 'foo' can be consumed using docker client

4. docker pull <pulp-hostname>:24817/foo

Required REST APIs

/v2/ - GET

/v2/<repository-name>/blobs/<digest> - HEAD, GET

/v2/<repository-name>/blobs/<digest>/uploads - POST

/v2/<repository-name>/blobs/<digest>/uploads/<uuid> - HEAD, GET, PUT, PATCH

/v2/<repository-name>/manifests/<digest or tag> - HEAD, GET, PUT

etherpad for reference for the endpoints https://etherpad.net/p/docker_push


Sub-issues 3 (0 open3 closed)

Story #6788: As an admin i can configure whether foreign layers can be pushed to the registryCLOSED - DUPLICATE

Actions
Task #6827: Ensure that all relations between content models are properly createdCLOSED - CURRENTRELEASE

Actions
Issue #6878: Auto create repos and distributions for the container pushCLOSED - CURRENTRELEASEipanova@redhat.comActions

Related issues

Related to Container Support - Story #4657: As a user i can use skopeo pulp transport to upload images to Pulp container registryCLOSED - WONTFIX

Actions
Actions #1

Updated by dkliban@redhat.com almost 5 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0

The POC of the above APIs are here[0]. However, this only creates the content in Pulp. No new repository versions are being created. These changes also require creating a Docker Distribution. However, this story intentionally omits having the user create a docker distribution to simplify the workflow.

[0] https://github.com/pulp/pulp_docker/compare/master...dkliban:docker-push?expand=1

Actions #2

Updated by dkliban@redhat.com almost 5 years ago

I updated the branch from the previous comment. However, it requires a change in pulpcore[0]. This change is currently only present on my fork of pulpcore[1].

The latest code creates a repository version on every push.

[0] https://pulp.plan.io/issues/5028
[1] https://github.com/dkliban/pulpcore/tree/5028-syncronous-repo-version-create

Actions #3

Updated by ipanova@redhat.com almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Sprint set to Sprint 55
Actions #4

Updated by ipanova@redhat.com almost 5 years ago

  • Description updated (diff)
Actions #5

Updated by ipanova@redhat.com almost 5 years ago

  • Project changed from Pulp to Docker Support
Actions #6

Updated by dkliban@redhat.com over 4 years ago

  • Sprint changed from Sprint 55 to Sprint 56
Actions #7

Updated by dkliban@redhat.com over 4 years ago

  • Status changed from ASSIGNED to NEW

The latest plan is to try and see if creating a temporary repository version for the duration of a docker push is a viable option. Then at the end of the push, when a tag is being created the repository version is created with a task.

Actions #8

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 56 to Sprint 57
Actions #9

Updated by dkliban@redhat.com over 4 years ago

During the last discussion about this feature we discussed a race condition. I'd like to discuss a possible solution.

Problem: docker push can result in corrupted repository if a blob, manifest, or manifest list is removed from the repository during the docker push workflow.

Solution: Perform validation of the content in the repository when the user makes the 'create tag' request. Return 400 error if any of the content is missing for that tag. This should check that the manifest list, manifest, and blobs are present. If a user receives a 400 error, she should try to push again. This time any content that went missing in the middle of the previous request will be re-uploaded.

This implies that we will create a new repository version with each blob/manifest/manifestlist/tag added to the repository. We would need to implement https://pulp.plan.io/issues/5028 as its description is currently written.

We could optionally squash repository versions when a tag is created. This would result in a single version for each tag added. This version would include all the manifests and blobks associated with the tag.

Actions #10

Updated by ipanova@redhat.com over 4 years ago

  • Related to Story #4657: As a user i can use skopeo pulp transport to upload images to Pulp container registry added
Actions #11

Updated by dkliban@redhat.com over 4 years ago

  • Assignee deleted (dkliban@redhat.com)
Actions #12

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 57 to Sprint 58
Actions #13

Updated by ipanova@redhat.com over 4 years ago

  • Sprint deleted (Sprint 58)
Actions #15

Updated by ipanova@redhat.com over 4 years ago

  • Project changed from Docker Support to Container Support
Actions #16

Updated by pulpbot almost 4 years ago

  • Status changed from NEW to POST

Added by dkliban@redhat.com almost 4 years ago

Revision 7f26e96a | View on GitHub

Adds REST APIs needed to perform docker/podman push

The user needs to create a Container Repository with a name that they will use to perform a push with. e.g. foo/bar

http :/pulp/api/v3/repositories/container/container/ name='foo/bar'

podman tag dhak235a localhost:24817/foo/bar podman push localhost:24817/foo/bar

Then the content can be pulled backed from Pulp by creating a distribution with the a base path that matches the name of the repository and has the repository associated with it. e.g. base_path='foo/bar' repository=

http :/pulp/api/v3/distributions/container/container/ name=foo base_path='foo/bar'
repository=/pulp/api/v3/repositories/container/container/1b5f0ac4-a911-42b8-bd36-4d497e6e9292/

podman pull localhost:24816/foo/bar

Note the difference in the port numbers of the two APIs.

re: #5027 https://pulp.plan.io/issues/5027

Added by dkliban@redhat.com almost 4 years ago

Revision 7f26e96a | View on GitHub

Adds REST APIs needed to perform docker/podman push

The user needs to create a Container Repository with a name that they will use to perform a push with. e.g. foo/bar

http :/pulp/api/v3/repositories/container/container/ name='foo/bar'

podman tag dhak235a localhost:24817/foo/bar podman push localhost:24817/foo/bar

Then the content can be pulled backed from Pulp by creating a distribution with the a base path that matches the name of the repository and has the repository associated with it. e.g. base_path='foo/bar' repository=

http :/pulp/api/v3/distributions/container/container/ name=foo base_path='foo/bar'
repository=/pulp/api/v3/repositories/container/container/1b5f0ac4-a911-42b8-bd36-4d497e6e9292/

podman pull localhost:24816/foo/bar

Note the difference in the port numbers of the two APIs.

re: #5027 https://pulp.plan.io/issues/5027

Actions #19

Updated by ipanova@redhat.com over 3 years ago

  • Status changed from POST to CLOSED - CURRENTRELEASE
Actions #20

Updated by daviddavis over 2 years ago

Looks like BZ 1313502 links to this issue.

Also available in: Atom PDF