Project

Profile

Help

Story #2993

As a user, I can upload a docker_manifest_list so that I can remove arches from a manifest list without performing a sync

Added by rmcgover 3 months ago. Updated 7 days ago.

Status:
POST
Priority:
Normal
Category:
-
Sprint/Milestone:
% Done:

0%

Platform Release:
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No

Description

I would like to be able to use Pulp's upload API to upload units of type docker_manifest_list.

Our use-case for this is removing selected arches from a manifest list.

For example: right now we can sync from a docker registry into Pulp and end up with a docker_manifest_list unit with content as in this sample:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux",
      }
    },
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:49ae93732fcf8d63fe1cce759664982dbd5b23161f007dba8561862adc96d063",
      "platform": {
        "architecture": "aarch64",
        "os": "linux",
      }
    },
    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7682,
      "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
      "platform": {
        "architecture": "amd64",
        "os": "linux",
        "features": [
          "sse4" 
        ]
      }
    }
  ]
}

The source used for sync included three arches. But, our requirement is that we want to be able to choose any combination of those arches for publish, thus we may need to create a new manifest list.

To get the new manifest list into Pulp, what I would like to do is:

  • Download manifest list from Pulp
  • Make a new manifest list by removing the items from "manifests" with an unwanted architecture
  • Upload modified manifest list to Pulp
  • Now I can remove the original manifest list and manifests for the arches I don't want, leaving only desired arches

If the upload isn't possible then we're instead forced to do a slower and more complicated process to get the same result indirectly: create a new manifest list, push it to a docker registry, ask Pulp to sync from that registry.

automate_upload.sh (1.03 KB) For QE: A script I used to test the REST API, amacdona@redhat.com, 11/14/2017 06:08 AM automate_upload.sh

Checklist

History

#1 Updated by rmcgover 3 months ago

Would someone from Pulp team be able to have a look at this? I'm not asking for a target release or date, just to know whether you agree or disagree with this request in principle. If you think it's a bad idea then let me know so we can plan around it.

#2 Updated by ipanova@redhat.com 2 months ago

This is doable, we just need to highlight the fact that the 'upload of a new manifest list' needs to contain image manifests which are already in pulp, or for those images manifests which are not in pulp, first upload them with skopeo utility into pulp. In other words the upload of a new manifest list would not imply the automatic upload of the image manifests+blobs referenced inside of the list.

#3 Updated by rmcgover 2 months ago

Yep, that's what I'd expect.

One thing I'm not sure is how far the validation should go; for example:

    {
      "mediaType": "application/vnd.docker.image.manifest.v2+json",
      "size": 7143,
      "digest": "sha256:49ae93732fcf8d63fe1cce759664982dbd5b23161f007dba8561862adc96d063",
      "platform": {
        "architecture": "aarch64",
        "os": "linux",
      }
    },

If I try to upload that in manifest list then should Pulp also check if that manifest matches the given size and platform, and reject (fatal error) if there's a mismatch?

From my point of view such checks would be welcome but not a hard requirement.

#4 Updated by ipanova@redhat.com about 2 months ago

I don't think we would validate this, it will completely up to the user to take care about the content he is uploading not lead to the corrupted manifest.
In addition we do not store in pulp size and platform information of the image manifest.

I guess maximum what we could validate is to check in the collection if the digest( since it's unique) of the image manifest listed within the manifest list is present and in case it is not reject/fail the upload.

#5 Updated by mhrivnak about 2 months ago

wrote:

I guess maximum what we could validate is to check in the collection if the digest( since it's unique) of the image manifest listed within the manifest list is present and in case it is not reject/fail the upload.

+1 I think that validation would be valuable and appropriate.

#6 Updated by ipanova@redhat.com about 1 month ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#7 Updated by jortel@redhat.com about 1 month ago

  • Sprint/Milestone set to Sprint 27

#8 Updated by amacdona@redhat.com 25 days ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

#9 Updated by mhrivnak 14 days ago

  • Sprint/Milestone changed from Sprint 27 to Sprint 28

#10 Updated by amacdona@redhat.com 7 days ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp_docker/pull/209

If it helps QE, I'll attach the bash script I used to test the REST API. (requires jq)

Please register to edit this issue

Also available in: Atom PDF