Story #3137
closedAs a user, I can view docker image Arch and Size Information through the REST API
100%
Description
As the result of this story it will be possible to see:
- arch and os of each image manifest listed within the manifest list queried trough Pulp API. Information will be added to the ManifestList model.
- size of each blob listed within the image manifest queried trough Pulp API. Information will be added to the Manifest model.
The information ( if available) will be taken from the manifest.json or manifest_list.json file returned from the registry.
Possible example of manifest list representation:
[
{
"updated": "2018-06-22T15:19:30Z",
"repo_id": "ina",
"created": "2018-06-22T15:19:30Z",
"unit_id": "06757e06-1c3c-4c4e-82d7-63fefb184935",
"unit_type_id": "docker_manifest_list",
"_id": {
"$oid": "5b2d1382c46f69d5b7f5258d"
},
"metadata": {
"_storage_path": "/var/lib/pulp/content/units/docker_manifest_list/43/a3683c58bce95e87474617e309bf1182a7a157c8d4616c44a8bf4ccc811e63/sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47",
"_id": "06757e06-1c3c-4c4e-82d7-63fefb184935",
"_ns": "units_docker_manifest_list",
"_last_updated": 1529680770,
"amd64_schema_version": 2,
"schema_version": 2,
"manifests": [
{"digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335",
"os": "linux",
"arch": "arm64"
},
{"digest": "sha256:35e28b647bd4976b7cacfaa32b7b253817d0881d77b6cda731ad46a29d08c2cb",
"os" :"linux",
"arch": "ppc64le"
},
{"digest: "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3",
"os": "linux",
"arch": "s390x"
}
],
"downloaded": true,
"pulp_user_metadata": {},
"_content_type_id": "docker_manifest_list",
"amd64_digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335",
"digest": "sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47"
}
}
]
Possible example of image manifest representation:
[
{
"updated": "2018-06-22T15:19:30Z",
"repo_id": "ina",
"created": "2018-06-22T15:19:30Z",
"unit_id": "0c966ae4-320d-43a0-a11c-42b3aa968824",
"unit_type_id": "docker_manifest",
"_id": {
"$oid": "5b2d1382c46f69d5b7f52590"
},
"metadata": {
"_storage_path": "/var/lib/pulp/content/units/docker_manifest/db/a1aa28355611292550bdfe7fcd7bcc793cc85892bffdb1ef1792c466960539/sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3",
"_ns": "units_docker_manifest",
"_last_updated": 1529680770,
"fs_layers": [
{
"layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"blob_sum": "sha256:eac115b78508fb249ed57f4e00662f90304fa86496413f393fdc1e2d5d793d83",
"size" : 1234567
}
],
"schema_version": 2,
"downloaded": true,
"pulp_user_metadata": {},
"_content_type_id": "docker_manifest",
"config_layer": "sha256:e6631e66b5519df3fb0ff6e5dc29b168f1d7adb9da3c04092f1894e15c2ca2b3",
"_id": "0c966ae4-320d-43a0-a11c-42b3aa968824",
"digest": "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3"
}
},
]
Note because none of the above fields( size, os, arch) are required by the registry API specification, they can be missing from the json file, that means - the values in Pulp are populated based on the availability of those in the json file returned from the registry.
For more information on what is defined on manifest list, image manifest schema2, image manifest schema1 check specs [0][1][2]
[0] https://docs.docker.com/registry/spec/manifest-v2-2/#example-manifest-list
[1] https://docs.docker.com/registry/spec/manifest-v2-2/#example-image-manifest
[2] https://docs.docker.com/registry/spec/manifest-v2-1/#example-manifest
Related issues
Updated by ipanova@redhat.com about 7 years ago
What are the reasons behind for this request?
Updated by ipanova@redhat.com about 7 years ago
when you say' blob requests' do you mean layers listed on the image manifests? because there is no information of blobs on the manifest list.
Updated by mhrivnak about 7 years ago
- Project changed from Pulp to Docker Support
- Subject changed from Make Docker Image Arch and Size Information API Accessible to As a user, I can view docker image Arch and Size Information through the REST API
- Category deleted (
14)
As @ipanova suggested, please provide each specific request that you are making to the API and what additional data each of the corresponding responses should have. That way we can be certain we understand this RFE correctly.
Depending on your needs, we may need to find a workaround at this late point in our release cycle.
Updated by breilly about 7 years ago
It pertains to this call: '/pulp/api/v2/repositories/<repo-id>/search/units/'
with the criteria being
'criteria': {
'type_ids': ['docker_manifest_list'],
'filters': {
'unit': {}
}
}
Right now this returns information about the manifest list, including the ids of every manifest in the manifest list. If arch information was added for each manifest it would be possible to keep track of which arch each manifest is intended for through the API, which as I understand it isn't possible right now.
By blob requests I meant that same call but with 'type_ids': ['docker_blob'] - this currently returns basic information about the blob, we'd like the size of the blob added to that output as well.
Updated by ipanova@redhat.com about 7 years ago
Thank you for providing information @breailly. I understand where these changes needs to go, but i still don't understand the reason
what is the purpose of keeping the track of specific arches?
what is the reason of the having the size of each blob?
Updated by breilly almost 7 years ago
Specific arch info is needed in the case of arch specific problems arising with available content - being able to see which manifests / blobs correspond to which arches via the API would allow for better diagnosis of the issue.
The size information was simply a 'nice to have'; it could be dropped from this request.
Updated by tomckay@redhat.com almost 7 years ago
I would like size information. In registry UI (foreman in my case) it would be useful to be able to see storage consumption. Users have asked for this size info to help them in choosing where to clean up disk space as well as to determine how much space is needed on a second registry (smart proxy).
Thanks!
Updated by ipanova@redhat.com over 6 years ago
- Is duplicate of Story #3450: As a user I would like to limit the tags we sync for docker repos added
Updated by ipanova@redhat.com over 6 years ago
- Is duplicate of deleted (Story #3450: As a user I would like to limit the tags we sync for docker repos)
Updated by ipanova@redhat.com over 6 years ago
on the manifest list model, besides the digest for each image manifest we will display its' arch and os( if those are available in the manifest_list.json file)
before:
"manifests" : [
"digest" : "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
"digest" : "sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28",
],
After:
"manifests" : [
{
"digest" : "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
"os" : "linux",
"arch" : "ppc64le"
},
{
"digest" : "sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28"
"os" : "linux",
"arch : "amd64"
}
],
Updated by ipanova@redhat.com over 6 years ago
blob model can have an additional field size, which will be populated with the result of os.stat()
> db.units_docker_blob.find()[0]
{
"_id" : "30df8e76-b8be-4183-be2e-6b64ceaa54a5",
"pulp_user_metadata" : {
},
"_last_updated" : 1525444105,
"_storage_path" : "/var/lib/pulp/content/units/docker_blob/86/0ac1a2efc27973c9cc967bd2c62fbab1355aa0574b8d8f7184935d80094195/sha256:daa414b19dcffa0ba7b80abb50a6b31156f1efd7bb1b8cdaeeb848da367afa38",
"downloaded" : true,
"digest" : "sha256:daa414b19dcffa0ba7b80abb50a6b31156f1efd7bb1b8cdaeeb848da367afa38",
"size" : 1760,
"_ns" : "units_docker_blob",
"_content_type_id" : "docker_blob"
or manifest model can have that info, but it would be extracted from manifest.json metadata file and it would look like:
> db.units_docker_manifest.find()[0]
{
"_id" : "6bd94c83-eb02-43d5-815a-27700d87e7e5",
"pulp_user_metadata" : {
},
"_last_updated" : 1525444105,
"_storage_path" : "/var/lib/pulp/content/units/docker_manifest/ba/7018fa3216c80b6c4e7f75580b7ff7f7d46c172f672ee2db006d9f6a0c757a/sha256:e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63",
"downloaded" : true,
"digest" : "sha256:e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63",
"schema_version" : 2,
"fs_layers" : [
{
"blob_sum" : "sha256:2a3ebcb7fbcc29bf40c4f62863008bb573acdea963454834d9483b3e5300c45d"
"size": 123455
}
],
"_ns" : "units_docker_manifest",
"_content_type_id" : "docker_manifest"
}
The problem about 2 approach is that shema1 does not have blob sizes on the manifest.json metadata filee at all, so we'd end up to have some manifests (like schema2) providing blob sizes from metadata and some(like schema1 not providing
We could implement both approaches where the size could be stored on the blob model after os.stat and where available( from manifest metadata json) on the manifest model as well.
Updated by ipanova@redhat.com over 6 years ago
based on some rmore eflection and discussion - we'll keep the size info of the blob on the blob model doing os.stat() before saving it into the db
Updated by ipanova@redhat.com over 6 years ago
- Sprint Candidate changed from No to Yes
Updated by dkliban@redhat.com over 6 years ago
- Groomed changed from No to Yes
- Sprint set to Sprint 39
Updated by dkliban@redhat.com over 6 years ago
- Has duplicate Story #2980: As QE I would like to have md5 checksums of docker_image, docker_blob units for correlation with other tools using md5 added
Updated by bizhang over 6 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to bizhang
Updated by rchan over 6 years ago
- Sprint/Milestone set to 2.17.0
Adding to 2.17.0 milestone. This is one of the required deliverables.
Updated by bizhang over 6 years ago
- Status changed from ASSIGNED to POST
Updated by dkliban@redhat.com over 6 years ago
- Sprint changed from Sprint 39 to Sprint 40
Added by werwty over 6 years ago
Added by werwty over 6 years ago
Revision ebcf64b8 | View on GitHub
Add arch, os field to ManifestList. Add size field to Manifest layers
Updated by werwty over 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset ebcf64b8b897d8f633d93295901986c0f5739326.
Added by werwty over 6 years ago
Revision baeef3ae | View on GitHub
Fix issue with associating embedded manifests in manifest list
Added by werwty over 6 years ago
Revision baeef3ae | View on GitHub
Fix issue with associating embedded manifests in manifest list
Updated by ipanova@redhat.com over 6 years ago
- Platform Release changed from 2.17.0 to master
Updated by ipanova@redhat.com over 6 years ago
- Platform Release deleted (
master)
Updated by ipanova@redhat.com over 6 years ago
- Status changed from MODIFIED to 5
Updated by ipanova@redhat.com over 6 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Add arch, os field to ManifestList. Add size field to Manifest layers
closes #3137 https://pulp.plan.io/issues/3137