Project

Profile

Help

Story #3137

As a user, I can view docker image Arch and Size Information through the REST API

Added by breilly over 1 year ago. Updated 5 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
% Done:

100%

Platform Release:
2.17.0
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 40

Description

As the result of this story it will be possible to see:

- arch and os of each image manifest listed within the manifest list queried trough Pulp API. Information will be added to the ManifestList model.
- size of each blob listed within the image manifest queried trough Pulp API. Information will be added to the Manifest model.

The information ( if available) will be taken from the manifest.json or manifest_list.json file returned from the registry.

Possible example of manifest list representation:

 [
  {
    "updated": "2018-06-22T15:19:30Z", 
    "repo_id": "ina", 
    "created": "2018-06-22T15:19:30Z", 
    "unit_id": "06757e06-1c3c-4c4e-82d7-63fefb184935", 
    "unit_type_id": "docker_manifest_list", 
    "_id": {
      "$oid": "5b2d1382c46f69d5b7f5258d" 
    }, 
    "metadata": {
      "_storage_path": "/var/lib/pulp/content/units/docker_manifest_list/43/a3683c58bce95e87474617e309bf1182a7a157c8d4616c44a8bf4ccc811e63/sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47", 
      "_id": "06757e06-1c3c-4c4e-82d7-63fefb184935", 
      "_ns": "units_docker_manifest_list", 
      "_last_updated": 1529680770, 
      "amd64_schema_version": 2, 
      "schema_version": 2, 
      "manifests": [
           {"digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335", 
            "os": "linux",
            "arch": "arm64" 
           },
           {"digest": "sha256:35e28b647bd4976b7cacfaa32b7b253817d0881d77b6cda731ad46a29d08c2cb", 
            "os" :"linux",
            "arch": "ppc64le" 
           },
           {"digest: "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3", 
            "os": "linux",
            "arch": "s390x" 
           }
      ], 
      "downloaded": true, 
      "pulp_user_metadata": {}, 
      "_content_type_id": "docker_manifest_list", 
      "amd64_digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335", 
      "digest": "sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47" 
    }
  }
]

Possible example of image manifest representation:

[
  {
    "updated": "2018-06-22T15:19:30Z", 
    "repo_id": "ina", 
    "created": "2018-06-22T15:19:30Z", 
    "unit_id": "0c966ae4-320d-43a0-a11c-42b3aa968824", 
    "unit_type_id": "docker_manifest", 
    "_id": {
      "$oid": "5b2d1382c46f69d5b7f52590" 
    }, 
    "metadata": {
      "_storage_path": "/var/lib/pulp/content/units/docker_manifest/db/a1aa28355611292550bdfe7fcd7bcc793cc85892bffdb1ef1792c466960539/sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3", 
      "_ns": "units_docker_manifest", 
      "_last_updated": 1529680770, 
      "fs_layers": [
        {
          "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
          "blob_sum": "sha256:eac115b78508fb249ed57f4e00662f90304fa86496413f393fdc1e2d5d793d83",
          "size" : 1234567
        }
      ], 
      "schema_version": 2, 
      "downloaded": true, 
      "pulp_user_metadata": {}, 
      "_content_type_id": "docker_manifest", 
      "config_layer": "sha256:e6631e66b5519df3fb0ff6e5dc29b168f1d7adb9da3c04092f1894e15c2ca2b3", 
      "_id": "0c966ae4-320d-43a0-a11c-42b3aa968824", 
      "digest": "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3" 
    }
  }, 
]

Note because none of the above fields( size, os, arch) are required by the registry API specification, they can be missing from the json file, that means - the values in Pulp are populated based on the availability of those in the json file returned from the registry.
For more information on what is defined on manifest list, image manifest schema2, image manifest schema1 check specs [0][1][2]
[0] https://docs.docker.com/registry/spec/manifest-v2-2/#example-manifest-list
[1] https://docs.docker.com/registry/spec/manifest-v2-2/#example-image-manifest
[2] https://docs.docker.com/registry/spec/manifest-v2-1/#example-manifest


Checklist


Related issues

Duplicated by Docker Support - Story #2980: As QE I would like to have md5 checksums of docker_image, docker_blob units for correlation with other tools using md5 CLOSED - DUPLICATE Actions

Associated revisions

Revision ebcf64b8 View on GitHub
Added by werwty 9 months ago

Add arch, os field to ManifestList. Add size field to Manifest layers

closes #3137
https://pulp.plan.io/issues/3137

Revision ebcf64b8 View on GitHub
Added by werwty 9 months ago

Add arch, os field to ManifestList. Add size field to Manifest layers

closes #3137
https://pulp.plan.io/issues/3137

Revision baeef3ae View on GitHub
Added by werwty 9 months ago

Fix issue with associating embedded manifests in manifest list

https://pulp.plan.io/issues/3892

re #3137
closes #3892

Revision baeef3ae View on GitHub
Added by werwty 9 months ago

Fix issue with associating embedded manifests in manifest list

https://pulp.plan.io/issues/3892

re #3137
closes #3892

History

#1 Updated by ipanova@redhat.com over 1 year ago

What are the reasons behind for this request?

#2 Updated by ipanova@redhat.com over 1 year ago

when you say' blob requests' do you mean layers listed on the image manifests? because there is no information of blobs on the manifest list.

#3 Updated by mhrivnak over 1 year ago

  • Project changed from Pulp to Docker Support
  • Subject changed from Make Docker Image Arch and Size Information API Accessible to As a user, I can view docker image Arch and Size Information through the REST API
  • Category deleted (API)

As @ipanova suggested, please provide each specific request that you are making to the API and what additional data each of the corresponding responses should have. That way we can be certain we understand this RFE correctly.

Depending on your needs, we may need to find a workaround at this late point in our release cycle.

#4 Updated by breilly over 1 year ago

It pertains to this call: '/pulp/api/v2/repositories/<repo-id>/search/units/'
with the criteria being

 'criteria': {                                                                        
    'type_ids': ['docker_manifest_list'],                   
    'filters': {                                                                     
        'unit': {}                                                                   
     }                                                                                
} 

Right now this returns information about the manifest list, including the ids of every manifest in the manifest list. If arch information was added for each manifest it would be possible to keep track of which arch each manifest is intended for through the API, which as I understand it isn't possible right now.

By blob requests I meant that same call but with 'type_ids': ['docker_blob'] - this currently returns basic information about the blob, we'd like the size of the blob added to that output as well.

#5 Updated by ipanova@redhat.com over 1 year ago

Thank you for providing information @breailly. I understand where these changes needs to go, but i still don't understand the reason

what is the purpose of keeping the track of specific arches?
what is the reason of the having the size of each blob?

#6 Updated by breilly over 1 year ago

Specific arch info is needed in the case of arch specific problems arising with available content - being able to see which manifests / blobs correspond to which arches via the API would allow for better diagnosis of the issue.

The size information was simply a 'nice to have'; it could be dropped from this request.

#7 Updated by tomckay@redhat.com over 1 year ago

I would like size information. In registry UI (foreman in my case) it would be useful to be able to see storage consumption. Users have asked for this size info to help them in choosing where to clean up disk space as well as to determine how much space is needed on a second registry (smart proxy).
Thanks!

#9 Updated by ipanova@redhat.com 12 months ago

  • Duplicates Story #3450: As a user I would like to limit the tags we sync for docker repos added

#10 Updated by ipanova@redhat.com 12 months ago

  • Duplicates deleted (Story #3450: As a user I would like to limit the tags we sync for docker repos)

#11 Updated by ipanova@redhat.com 12 months ago

on the manifest list model, besides the digest for each image manifest we will display its' arch and os( if those are available in the manifest_list.json file)

before:

"manifests" : [
            "digest" : "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
            "digest" : "sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28",

    ],

After:


"manifests" : [
        {
            "digest" : "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
            "os" : "linux",
            "arch" : "ppc64le" 
        },
        {
            "digest" : "sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28" 
            "os" : "linux",
            "arch : "amd64" 
        }
    ],

#12 Updated by ipanova@redhat.com 12 months ago

blob model can have an additional field size, which will be populated with the result of os.stat()


 > db.units_docker_blob.find()[0]
{
    "_id" : "30df8e76-b8be-4183-be2e-6b64ceaa54a5",
    "pulp_user_metadata" : {

    },
    "_last_updated" : 1525444105,
    "_storage_path" : "/var/lib/pulp/content/units/docker_blob/86/0ac1a2efc27973c9cc967bd2c62fbab1355aa0574b8d8f7184935d80094195/sha256:daa414b19dcffa0ba7b80abb50a6b31156f1efd7bb1b8cdaeeb848da367afa38",
    "downloaded" : true,
    "digest" : "sha256:daa414b19dcffa0ba7b80abb50a6b31156f1efd7bb1b8cdaeeb848da367afa38",
    "size" : 1760,
    "_ns" : "units_docker_blob",
    "_content_type_id" : "docker_blob" 

or manifest model can have that info, but it would be extracted from manifest.json metadata file and it would look like:

> db.units_docker_manifest.find()[0]
{
    "_id" : "6bd94c83-eb02-43d5-815a-27700d87e7e5",
    "pulp_user_metadata" : {

    },
    "_last_updated" : 1525444105,
    "_storage_path" : "/var/lib/pulp/content/units/docker_manifest/ba/7018fa3216c80b6c4e7f75580b7ff7f7d46c172f672ee2db006d9f6a0c757a/sha256:e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63",
    "downloaded" : true,
    "digest" : "sha256:e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63",
    "schema_version" : 2,
    "fs_layers" : [
        {
            "blob_sum" : "sha256:2a3ebcb7fbcc29bf40c4f62863008bb573acdea963454834d9483b3e5300c45d" 
            "size": 123455
        }
    ],
    "_ns" : "units_docker_manifest",
    "_content_type_id" : "docker_manifest" 
}

The problem about 2 approach is that shema1 does not have blob sizes on the manifest.json metadata filee at all, so we'd end up to have some manifests (like schema2) providing blob sizes from metadata and some(like schema1 not providing

We could implement both approaches where the size could be stored on the blob model after os.stat and where available( from manifest metadata json) on the manifest model as well.

#13 Updated by ipanova@redhat.com 12 months ago

based on some rmore eflection and discussion - we'll keep the size info of the blob on the blob model doing os.stat() before saving it into the db

#14 Updated by ipanova@redhat.com 10 months ago

  • Sprint Candidate changed from No to Yes

#15 Updated by ipanova@redhat.com 10 months ago

  • Description updated (diff)

#16 Updated by ipanova@redhat.com 10 months ago

  • Description updated (diff)

#17 Updated by dkliban@redhat.com 10 months ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 39

#18 Updated by dkliban@redhat.com 10 months ago

  • Duplicated by Story #2980: As QE I would like to have md5 checksums of docker_image, docker_blob units for correlation with other tools using md5 added

#19 Updated by bizhang 10 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang

#20 Updated by rchan 10 months ago

  • Sprint/Milestone set to 2.17.0

Adding to 2.17.0 milestone. This is one of the required deliverables.

#21 Updated by bizhang 9 months ago

  • Status changed from ASSIGNED to POST

#22 Updated by dkliban@redhat.com 9 months ago

  • Sprint changed from Sprint 39 to Sprint 40

#23 Updated by werwty 9 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#24 Updated by ipanova@redhat.com 9 months ago

  • Platform Release set to 2.17.0

#25 Updated by ipanova@redhat.com 9 months ago

  • Platform Release changed from 2.17.0 to master

#26 Updated by ipanova@redhat.com 9 months ago

  • Platform Release deleted (master)

#27 Updated by ipanova@redhat.com 9 months ago

  • Platform Release set to 2.17.0

#28 Updated by daviddavis 8 months ago

  • Smash Test set to 17

#29 Updated by ipanova@redhat.com 8 months ago

  • Status changed from MODIFIED to ON_QA

#30 Updated by ipanova@redhat.com 8 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#31 Updated by bmbouter 5 days ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF