Project

Profile

Help

Story #3137

closed

As a user, I can view docker image Arch and Size Information through the REST API

Added by breilly about 7 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.17.0
Target Release - Docker:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 40
Quarter:

Description

As the result of this story it will be possible to see:

- arch and os of each image manifest listed within the manifest list queried trough Pulp API. Information will be added to the ManifestList model.
- size of each blob listed within the image manifest queried trough Pulp API. Information will be added to the Manifest model.

The information ( if available) will be taken from the manifest.json or manifest_list.json file returned from the registry.

Possible example of manifest list representation:

 [
  {
    "updated": "2018-06-22T15:19:30Z", 
    "repo_id": "ina", 
    "created": "2018-06-22T15:19:30Z", 
    "unit_id": "06757e06-1c3c-4c4e-82d7-63fefb184935", 
    "unit_type_id": "docker_manifest_list", 
    "_id": {
      "$oid": "5b2d1382c46f69d5b7f5258d"
    }, 
    "metadata": {
      "_storage_path": "/var/lib/pulp/content/units/docker_manifest_list/43/a3683c58bce95e87474617e309bf1182a7a157c8d4616c44a8bf4ccc811e63/sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47", 
      "_id": "06757e06-1c3c-4c4e-82d7-63fefb184935", 
      "_ns": "units_docker_manifest_list", 
      "_last_updated": 1529680770, 
      "amd64_schema_version": 2, 
      "schema_version": 2, 
      "manifests": [
           {"digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335", 
            "os": "linux",
            "arch": "arm64"
           },
           {"digest": "sha256:35e28b647bd4976b7cacfaa32b7b253817d0881d77b6cda731ad46a29d08c2cb", 
            "os" :"linux",
            "arch": "ppc64le"
           },
           {"digest: "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3", 
            "os": "linux",
            "arch": "s390x"
           }
      ], 
      "downloaded": true, 
      "pulp_user_metadata": {}, 
      "_content_type_id": "docker_manifest_list", 
      "amd64_digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335", 
      "digest": "sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47"
    }
  }
]

Possible example of image manifest representation:

[
  {
    "updated": "2018-06-22T15:19:30Z", 
    "repo_id": "ina", 
    "created": "2018-06-22T15:19:30Z", 
    "unit_id": "0c966ae4-320d-43a0-a11c-42b3aa968824", 
    "unit_type_id": "docker_manifest", 
    "_id": {
      "$oid": "5b2d1382c46f69d5b7f52590"
    }, 
    "metadata": {
      "_storage_path": "/var/lib/pulp/content/units/docker_manifest/db/a1aa28355611292550bdfe7fcd7bcc793cc85892bffdb1ef1792c466960539/sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3", 
      "_ns": "units_docker_manifest", 
      "_last_updated": 1529680770, 
      "fs_layers": [
        {
          "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
          "blob_sum": "sha256:eac115b78508fb249ed57f4e00662f90304fa86496413f393fdc1e2d5d793d83",
          "size" : 1234567
        }
      ], 
      "schema_version": 2, 
      "downloaded": true, 
      "pulp_user_metadata": {}, 
      "_content_type_id": "docker_manifest", 
      "config_layer": "sha256:e6631e66b5519df3fb0ff6e5dc29b168f1d7adb9da3c04092f1894e15c2ca2b3", 
      "_id": "0c966ae4-320d-43a0-a11c-42b3aa968824", 
      "digest": "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3"
    }
  }, 
]

Note because none of the above fields( size, os, arch) are required by the registry API specification, they can be missing from the json file, that means - the values in Pulp are populated based on the availability of those in the json file returned from the registry.
For more information on what is defined on manifest list, image manifest schema2, image manifest schema1 check specs [0][1][2]
[0] https://docs.docker.com/registry/spec/manifest-v2-2/#example-manifest-list
[1] https://docs.docker.com/registry/spec/manifest-v2-2/#example-image-manifest
[2] https://docs.docker.com/registry/spec/manifest-v2-1/#example-manifest


Related issues

Has duplicate Docker Support - Story #2980: As QE I would like to have md5 checksums of docker_image, docker_blob units for correlation with other tools using md5CLOSED - DUPLICATE

Actions
Actions #1

Updated by ipanova@redhat.com about 7 years ago

What are the reasons behind for this request?

Actions #2

Updated by ipanova@redhat.com about 7 years ago

when you say' blob requests' do you mean layers listed on the image manifests? because there is no information of blobs on the manifest list.

Actions #3

Updated by mhrivnak about 7 years ago

  • Project changed from Pulp to Docker Support
  • Subject changed from Make Docker Image Arch and Size Information API Accessible to As a user, I can view docker image Arch and Size Information through the REST API
  • Category deleted (14)

As @ipanova suggested, please provide each specific request that you are making to the API and what additional data each of the corresponding responses should have. That way we can be certain we understand this RFE correctly.

Depending on your needs, we may need to find a workaround at this late point in our release cycle.

Actions #4

Updated by breilly about 7 years ago

It pertains to this call: '/pulp/api/v2/repositories/<repo-id>/search/units/'
with the criteria being

 'criteria': {                                                                        
    'type_ids': ['docker_manifest_list'],                   
    'filters': {                                                                     
        'unit': {}                                                                   
     }                                                                                
} 

Right now this returns information about the manifest list, including the ids of every manifest in the manifest list. If arch information was added for each manifest it would be possible to keep track of which arch each manifest is intended for through the API, which as I understand it isn't possible right now.

By blob requests I meant that same call but with 'type_ids': ['docker_blob'] - this currently returns basic information about the blob, we'd like the size of the blob added to that output as well.

Actions #5

Updated by ipanova@redhat.com about 7 years ago

Thank you for providing information @breailly. I understand where these changes needs to go, but i still don't understand the reason

what is the purpose of keeping the track of specific arches?
what is the reason of the having the size of each blob?

Actions #6

Updated by breilly almost 7 years ago

Specific arch info is needed in the case of arch specific problems arising with available content - being able to see which manifests / blobs correspond to which arches via the API would allow for better diagnosis of the issue.

The size information was simply a 'nice to have'; it could be dropped from this request.

Actions #7

Updated by tomckay@redhat.com almost 7 years ago

I would like size information. In registry UI (foreman in my case) it would be useful to be able to see storage consumption. Users have asked for this size info to help them in choosing where to clean up disk space as well as to determine how much space is needed on a second registry (smart proxy).
Thanks!

Actions #9

Updated by ipanova@redhat.com over 6 years ago

  • Is duplicate of Story #3450: As a user I would like to limit the tags we sync for docker repos added
Actions #10

Updated by ipanova@redhat.com over 6 years ago

  • Is duplicate of deleted (Story #3450: As a user I would like to limit the tags we sync for docker repos)
Actions #11

Updated by ipanova@redhat.com over 6 years ago

on the manifest list model, besides the digest for each image manifest we will display its' arch and os( if those are available in the manifest_list.json file)

before:

"manifests" : [
            "digest" : "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
            "digest" : "sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28",

    ],

After:


"manifests" : [
        {
            "digest" : "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
            "os" : "linux",
            "arch" : "ppc64le"
        },
        {
            "digest" : "sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28"
            "os" : "linux",
            "arch : "amd64"
        }
    ],
Actions #12

Updated by ipanova@redhat.com over 6 years ago

blob model can have an additional field size, which will be populated with the result of os.stat()


 > db.units_docker_blob.find()[0]
{
    "_id" : "30df8e76-b8be-4183-be2e-6b64ceaa54a5",
    "pulp_user_metadata" : {

    },
    "_last_updated" : 1525444105,
    "_storage_path" : "/var/lib/pulp/content/units/docker_blob/86/0ac1a2efc27973c9cc967bd2c62fbab1355aa0574b8d8f7184935d80094195/sha256:daa414b19dcffa0ba7b80abb50a6b31156f1efd7bb1b8cdaeeb848da367afa38",
    "downloaded" : true,
    "digest" : "sha256:daa414b19dcffa0ba7b80abb50a6b31156f1efd7bb1b8cdaeeb848da367afa38",
    "size" : 1760,
    "_ns" : "units_docker_blob",
    "_content_type_id" : "docker_blob"

or manifest model can have that info, but it would be extracted from manifest.json metadata file and it would look like:

> db.units_docker_manifest.find()[0]
{
    "_id" : "6bd94c83-eb02-43d5-815a-27700d87e7e5",
    "pulp_user_metadata" : {

    },
    "_last_updated" : 1525444105,
    "_storage_path" : "/var/lib/pulp/content/units/docker_manifest/ba/7018fa3216c80b6c4e7f75580b7ff7f7d46c172f672ee2db006d9f6a0c757a/sha256:e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63",
    "downloaded" : true,
    "digest" : "sha256:e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63",
    "schema_version" : 2,
    "fs_layers" : [
        {
            "blob_sum" : "sha256:2a3ebcb7fbcc29bf40c4f62863008bb573acdea963454834d9483b3e5300c45d"
            "size": 123455
        }
    ],
    "_ns" : "units_docker_manifest",
    "_content_type_id" : "docker_manifest"
}

The problem about 2 approach is that shema1 does not have blob sizes on the manifest.json metadata filee at all, so we'd end up to have some manifests (like schema2) providing blob sizes from metadata and some(like schema1 not providing

We could implement both approaches where the size could be stored on the blob model after os.stat and where available( from manifest metadata json) on the manifest model as well.

Actions #13

Updated by ipanova@redhat.com over 6 years ago

based on some rmore eflection and discussion - we'll keep the size info of the blob on the blob model doing os.stat() before saving it into the db

Actions #14

Updated by ipanova@redhat.com over 6 years ago

  • Sprint Candidate changed from No to Yes
Actions #15

Updated by ipanova@redhat.com over 6 years ago

  • Description updated (diff)
Actions #16

Updated by ipanova@redhat.com over 6 years ago

  • Description updated (diff)
Actions #17

Updated by dkliban@redhat.com over 6 years ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 39
Actions #18

Updated by dkliban@redhat.com over 6 years ago

  • Has duplicate Story #2980: As QE I would like to have md5 checksums of docker_image, docker_blob units for correlation with other tools using md5 added
Actions #19

Updated by bizhang over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang
Actions #20

Updated by rchan over 6 years ago

  • Sprint/Milestone set to 2.17.0

Adding to 2.17.0 milestone. This is one of the required deliverables.

Actions #21

Updated by bizhang over 6 years ago

  • Status changed from ASSIGNED to POST
Actions #22

Updated by dkliban@redhat.com over 6 years ago

  • Sprint changed from Sprint 39 to Sprint 40

Added by werwty over 6 years ago

Revision ebcf64b8 | View on GitHub

Add arch, os field to ManifestList. Add size field to Manifest layers

closes #3137 https://pulp.plan.io/issues/3137

Added by werwty over 6 years ago

Revision ebcf64b8 | View on GitHub

Add arch, os field to ManifestList. Add size field to Manifest layers

closes #3137 https://pulp.plan.io/issues/3137

Actions #23

Updated by werwty over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Added by werwty over 6 years ago

Revision baeef3ae | View on GitHub

Fix issue with associating embedded manifests in manifest list

https://pulp.plan.io/issues/3892

re #3137 closes #3892

Added by werwty over 6 years ago

Revision baeef3ae | View on GitHub

Fix issue with associating embedded manifests in manifest list

https://pulp.plan.io/issues/3892

re #3137 closes #3892

Actions #24

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release set to 2.17.0
Actions #25

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release changed from 2.17.0 to master
Actions #26

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release deleted (master)
Actions #27

Updated by ipanova@redhat.com over 6 years ago

  • Platform Release set to 2.17.0
Actions #29

Updated by ipanova@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5
Actions #30

Updated by ipanova@redhat.com over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #31

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions #33

Updated by ttereshc almost 3 years ago

  • Sprint/Milestone deleted (2.17.0)

Also available in: Atom PDF