Project

Profile

Help

Actions
Send by e-mail Copy link

Story #2993

closed

As a user, I can upload a docker_manifest_list so that I can remove arches from a manifest list without performing a sync

Added by rmcgover over 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
amacdona@redhat.com
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.15.0
Target Release - Docker:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 28
Quarter:

Description

I would like to be able to use Pulp's upload API to upload units of type docker_manifest_list.

Our use-case for this is removing selected arches from a manifest list.

For example: right now we can sync from a docker registry into Pulp and end up with a docker_manifest_list unit with content as in this sample:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux",
      }
    },
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:49ae93732fcf8d63fe1cce759664982dbd5b23161f007dba8561862adc96d063",
      "platform": {
        "architecture": "aarch64",
        "os": "linux",
      }
    },
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7682,
      "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
      "platform": {
        "architecture": "amd64",
        "os": "linux",
        "features": [
          "sse4"
        ]
      }
    }
  ]
}

The source used for sync included three arches. But, our requirement is that we want to be able to choose any combination of those arches for publish, thus we may need to create a new manifest list.

To get the new manifest list into Pulp, what I would like to do is:

  • Download manifest list from Pulp
  • Make a new manifest list by removing the items from "manifests" with an unwanted architecture
  • Upload modified manifest list to Pulp
  • Now I can remove the original manifest list and manifests for the arches I don't want, leaving only desired arches

If the upload isn't possible then we're instead forced to do a slower and more complicated process to get the same result indirectly: create a new manifest list, push it to a docker registry, ask Pulp to sync from that registry.

Files

automate_upload.sh (1.03 KB) automate_upload.sh For QE: A script I used to test the REST API, amacdona@redhat.com, 11/14/2017 06:08 AM
Actions
Copy link
 #1

Updated by rmcgover over 5 years ago

Would someone from Pulp team be able to have a look at this? I'm not asking for a target release or date, just to know whether you agree or disagree with this request in principle. If you think it's a bad idea then let me know so we can plan around it.

Actions
Copy link
 #2

Updated by ipanova@redhat.com over 5 years ago

This is doable, we just need to highlight the fact that the 'upload of a new manifest list' needs to contain image manifests which are already in pulp, or for those images manifests which are not in pulp, first upload them with skopeo utility into pulp. In other words the upload of a new manifest list would not imply the automatic upload of the image manifests+blobs referenced inside of the list.

Actions
Copy link
 #3

Updated by rmcgover over 5 years ago

Yep, that's what I'd expect.

One thing I'm not sure is how far the validation should go; for example:

    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:49ae93732fcf8d63fe1cce759664982dbd5b23161f007dba8561862adc96d063",
      "platform": {
        "architecture": "aarch64",
        "os": "linux",
      }
    },

If I try to upload that in manifest list then should Pulp also check if that manifest matches the given size and platform, and reject (fatal error) if there's a mismatch?

From my point of view such checks would be welcome but not a hard requirement.

Actions
Copy link
 #4

Updated by ipanova@redhat.com over 5 years ago

I don't think we would validate this, it will completely up to the user to take care about the content he is uploading not lead to the corrupted manifest.
In addition we do not store in pulp size and platform information of the image manifest.

I guess maximum what we could validate is to check in the collection if the digest( since it's unique) of the image manifest listed within the manifest list is present and in case it is not reject/fail the upload.

Actions
Copy link
 #5

Updated by mhrivnak over 5 years ago

wrote:

I guess maximum what we could validate is to check in the collection if the digest( since it's unique) of the image manifest listed within the manifest list is present and in case it is not reject/fail the upload.

+1 I think that validation would be valuable and appropriate.

Actions
Copy link
 #6

Updated by ipanova@redhat.com over 5 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions
Copy link
 #7

Updated by jortel@redhat.com over 5 years ago

  • Sprint/Milestone set to 46
Actions
Copy link
 #8

Updated by amacdona@redhat.com over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com
Actions
Copy link
 #9

Updated by mhrivnak over 5 years ago

  • Sprint/Milestone changed from 46 to 47
Actions
Copy link
 #10

Updated by amacdona@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp_docker/pull/209

If it helps QE, I'll attach the bash script I used to test the REST API. (requires jq)

Added by amacdona@redhat.com over 5 years ago

Revision cc6f1c52

Implement uploads for Docker Manifest List

Manifest lists can be uploaded to a repository IFF all of the referenced Image Manifests are already associated to the repository.

closes #2993

Added by amacdona@redhat.com over 5 years ago

Revision cc6f1c52

Implement uploads for Docker Manifest List

Manifest lists can be uploaded to a repository IFF all of the referenced Image Manifests are already associated to the repository.

closes #2993

Added by amacdona@redhat.com over 5 years ago

Revision cc6f1c52

Implement uploads for Docker Manifest List

Manifest lists can be uploaded to a repository IFF all of the referenced Image Manifests are already associated to the repository.

closes #2993

Added by amacdona@redhat.com over 5 years ago

Revision cc6f1c52

Implement uploads for Docker Manifest List

Manifest lists can be uploaded to a repository IFF all of the referenced Image Manifests are already associated to the repository.

closes #2993

Actions
Copy link
 #12

Updated by amacdona@redhat.com over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #13

Updated by pcreech over 5 years ago

  • Platform Release set to 2.15.0
Actions
Copy link
 #14

Updated by pcreech over 5 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #15

Updated by rmcgover over 5 years ago

In the example given with this bug report, the mimetype appearing within manifests seems to be wrong - application/vnd.docker.image.manifest.v2+json was used but apparently it should be application/vnd.docker.distribution.manifest.v2+json.

Looks like that's from a bug in the docs for docker/distribution which Ina has already found and fixed in https://github.com/docker/distribution/commit/5ccd03d28ae2b23a3b2863216bcb97e9f650f6d2#diff-298c8e761320ff3dde0420467054f000 .

Actions
Copy link
 #16

Updated by Ichimonji10 over 5 years ago

@preethi I think this issue can be automated.

Actions
Copy link
 #18

Updated by pcreech about 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #19

Updated by bmbouter about 5 years ago

  • Sprint set to Sprint 28
Actions
Copy link
 #20

Updated by bmbouter about 5 years ago

  • Sprint/Milestone deleted (47)
Actions
Copy link
 #21

Updated by bmbouter almost 4 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF