Project

Profile

Help

Story #2993

As a user, I can upload a docker_manifest_list so that I can remove arches from a manifest list without performing a sync

Added by rmcgover 5 months ago. Updated 4 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
% Done:

100%

Platform Release:
2.15.0
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No

Description

I would like to be able to use Pulp's upload API to upload units of type docker_manifest_list.

Our use-case for this is removing selected arches from a manifest list.

For example: right now we can sync from a docker registry into Pulp and end up with a docker_manifest_list unit with content as in this sample:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux",
      }
    },
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:49ae93732fcf8d63fe1cce759664982dbd5b23161f007dba8561862adc96d063",
      "platform": {
        "architecture": "aarch64",
        "os": "linux",
      }
    },
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7682,
      "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
      "platform": {
        "architecture": "amd64",
        "os": "linux",
        "features": [
          "sse4" 
        ]
      }
    }
  ]
}

The source used for sync included three arches. But, our requirement is that we want to be able to choose any combination of those arches for publish, thus we may need to create a new manifest list.

To get the new manifest list into Pulp, what I would like to do is:

  • Download manifest list from Pulp
  • Make a new manifest list by removing the items from "manifests" with an unwanted architecture
  • Upload modified manifest list to Pulp
  • Now I can remove the original manifest list and manifests for the arches I don't want, leaving only desired arches

If the upload isn't possible then we're instead forced to do a slower and more complicated process to get the same result indirectly: create a new manifest list, push it to a docker registry, ask Pulp to sync from that registry.

automate_upload.sh (1.03 KB) For QE: A script I used to test the REST API, amacdona@redhat.com, 11/14/2017 06:08 AM automate_upload.sh

Checklist

Associated revisions

Revision cc6f1c52 View on GitHub
Added by amacdona@redhat.com about 2 months ago

Implement uploads for Docker Manifest List

Manifest lists can be uploaded to a repository IFF all of the
referenced Image Manifests are already associated to the repository.

closes #2993

History

#1 Updated by rmcgover 5 months ago

Would someone from Pulp team be able to have a look at this? I'm not asking for a target release or date, just to know whether you agree or disagree with this request in principle. If you think it's a bad idea then let me know so we can plan around it.

#2 Updated by ipanova@redhat.com 4 months ago

This is doable, we just need to highlight the fact that the 'upload of a new manifest list' needs to contain image manifests which are already in pulp, or for those images manifests which are not in pulp, first upload them with skopeo utility into pulp. In other words the upload of a new manifest list would not imply the automatic upload of the image manifests+blobs referenced inside of the list.

#3 Updated by rmcgover 4 months ago

Yep, that's what I'd expect.

One thing I'm not sure is how far the validation should go; for example:

    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:49ae93732fcf8d63fe1cce759664982dbd5b23161f007dba8561862adc96d063",
      "platform": {
        "architecture": "aarch64",
        "os": "linux",
      }
    },

If I try to upload that in manifest list then should Pulp also check if that manifest matches the given size and platform, and reject (fatal error) if there's a mismatch?

From my point of view such checks would be welcome but not a hard requirement.

#4 Updated by ipanova@redhat.com 4 months ago

I don't think we would validate this, it will completely up to the user to take care about the content he is uploading not lead to the corrupted manifest.
In addition we do not store in pulp size and platform information of the image manifest.

I guess maximum what we could validate is to check in the collection if the digest( since it's unique) of the image manifest listed within the manifest list is present and in case it is not reject/fail the upload.

#5 Updated by mhrivnak 4 months ago

wrote:

I guess maximum what we could validate is to check in the collection if the digest( since it's unique) of the image manifest listed within the manifest list is present and in case it is not reject/fail the upload.

+1 I think that validation would be valuable and appropriate.

#6 Updated by ipanova@redhat.com 3 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#7 Updated by jortel@redhat.com 3 months ago

  • Sprint/Milestone set to Sprint 27

#8 Updated by amacdona@redhat.com 3 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

#9 Updated by mhrivnak 3 months ago

  • Sprint/Milestone changed from Sprint 27 to Sprint 28

#10 Updated by amacdona@redhat.com 2 months ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp_docker/pull/209

If it helps QE, I'll attach the bash script I used to test the REST API. (requires jq)

#12 Updated by amacdona@redhat.com about 2 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#13 Updated by pcreech about 2 months ago

  • Platform Release set to 2.15.0

#14 Updated by pcreech about 2 months ago

  • Status changed from MODIFIED to ON_QA

#15 Updated by rmcgover about 2 months ago

In the example given with this bug report, the mimetype appearing within manifests seems to be wrong - application/vnd.docker.image.manifest.v2+json was used but apparently it should be application/vnd.docker.distribution.manifest.v2+json.

Looks like that's from a bug in the docs for docker/distribution which Ina has already found and fixed in https://github.com/docker/distribution/commit/5ccd03d28ae2b23a3b2863216bcb97e9f650f6d2#diff-298c8e761320ff3dde0420467054f000 .

#16 Updated by Ichimonji10 about 1 month ago

@preethi I think this issue can be automated.

#17 Updated by pthomas@redhat.com about 1 month ago

  • Smash Test set to 829

#18 Updated by pcreech 4 days ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF