Project

Profile

Help

Issue #6814

closed

docker-content-digest header is missing from the response on s3 files

Added by ipanova@redhat.com almost 4 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 75
Quarter:

Description

we have set content-type and content-disposition headers, but we also need docker-content-digest present among response headers

usecase: Users want to make a HEAD request for a tag to find out if that tag is pointing to a new manifest. Without this header, users must download the manifest in order to learn its digest.

Actions #1

Updated by ipanova@redhat.com almost 4 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 73
Actions #2

Updated by mdellweg almost 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mdellweg
Actions #3

Updated by mdellweg almost 4 years ago

A little bit of poking into the registry api at quay lets me to believe that is is sufficient to hand over the Docker-Content-Digest header with the redirect response (assuming they have found a proper way to handle it):

$ http --header https://quay.io/v2/pulp/pulp-fixtures/blobs/sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5
HTTP/1.1 302 FOUND
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Connection: close
Content-Length: 1225
Content-Type: text/html; charset=utf-8
Date: Tue, 26 May 2020 11:21:26 GMT
Docker-Content-Digest: sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5
Location: https://cdn.quay.io/sha256/55/55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5?Expires=1590492686&Signature=MttvuW56~Obzu-t5lz6w8Xx8BFYMeWX3NekRdNHm0aGS1dncw7lBo5uzUB1LKtXxWM3gsK31-StT56rp9N05cKCzvNpw1oYrPslLdiLBBErexw~DJwYrDENheza-2AfO045IN7PS20NejpTTece55srCkXBRjZvdMM4N5AKSkFfVORST4AfPjaWjQenB9wyHQgca9Ux3EmK2UsYAhb953rjBA5sdo6MZV-I87CYHwCggsOwioG33PuGZV5LhgXGpjOUeA1QQPBE1a3M-MsvQ5bvvsWMinYRA7nPsnhyihMfomrL2VtTTyG7foeLAM0rGdvLw7BOiz7faT-TiNPUQSw__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA
Server: nginx/1.12.1
Strict-Transport-Security: max-age=63072000; preload
X-Frame-Options: DENY
$ http --header "https://cdn.quay.io/sha256/55/55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5?Expires=1590492686&Signature=MttvuW56~Obzu-t5lz6w8Xx8BFYMeWX3NekRdNHm0aGS1dncw7lBo5uzUB1LKtXxWM3gsK31-StT56rp9N05cKCzvNpw1oYrPslLdiLBBErexw~DJwYrDENheza-2AfO045IN7PS20NejpTTece55srCkXBRjZvdMM4N5AKSkFfVORST4AfPjaWjQenB9wyHQgca9Ux3EmK2UsYAhb953rjBA5sdo6MZV-I87CYHwCggsOwioG33PuGZV5LhgXGpjOUeA1QQPBE1a3M-MsvQ5bvvsWMinYRA7nPsnhyihMfomrL2VtTTyG7foeLAM0rGdvLw7BOiz7faT-TiNPUQSw__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA"
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: keep-alive
Content-Length: 967332469
Content-Type: binary/octet-stream
Date: Tue, 26 May 2020 11:22:57 GMT
ETag: "1f0c077b3d78c90bcb5f626336fa1f0f-1"
Last-Modified: Tue, 26 May 2020 09:49:00 GMT
Server: AmazonS3
Via: 1.1 77d8cf253666facea1bbe67902fcbbc1.cloudfront.net (CloudFront)
X-Amz-Cf-Id: p9yL-6-NU_zyRfW6Q_6RCtir33llG4e2HyfIx3OK4cPdQvQ9TGymZA==
X-Amz-Cf-Pop: DUS51-C1
X-Cache: Miss from cloudfront
x-amz-replication-status: COMPLETED
x-amz-server-side-encryption: AES256
x-amz-version-id: kl.cHTDXPhTMMv50HTFct7Z7VazmSMG7
Actions #4

Updated by ipanova@redhat.com almost 4 years ago

I also have noticed that when fetching manfest a 200 is given and not 302 (GET/HEAD)

$ curl -i -X HEAD https://quay.io/v2/pulp/pulp-fixtures/manifests/latest -H "Accept: application/vnd.docker.distribution.manifest.v2+json"
HTTP/1.1 200 OK
Server: nginx/1.12.1
Date: Tue, 26 May 2020 12:29:42 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1784
Connection: close
Docker-Content-Digest: sha256:90c026a04b412530e76d96e3ec57f79775275b5ef2643cac8e867b111f739d82
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload


(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ curl -i -X HEAD https://quay.io/v2/pulp/pulp-fixtures/manifests/sha256:90c026a04b412530e76d96e3ec57f79775275b5ef2643cac8e867b111f739d82 -H "Accept: application/vnd.docker.distribution.manifest.v2+json"
HTTP/1.1 200 OK
Server: nginx/1.12.1
Date: Tue, 26 May 2020 12:29:59 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1784
Connection: close
Docker-Content-Digest: sha256:90c026a04b412530e76d96e3ec57f79775275b5ef2643cac8e867b111f739d82
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload


$ curl -i -X GET https://quay.io/v2/pulp/pulp-fixtures/manifests/latest -H "Accept: application/vnd.docker.distribution.manifest.v2+json"
HTTP/1.1 200 OK
Server: nginx/1.12.1
Date: Tue, 26 May 2020 15:19:16 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1784
Connection: close
Docker-Content-Digest: sha256:90c026a04b412530e76d96e3ec57f79775275b5ef2643cac8e867b111f739d82
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 7848,
      "digest": "sha256:8e8b4fd66a958bbaa0be54c73b746bdb84b615982445713ff8362a35d11c7db2"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 27098756,
         "digest": "sha256:afb6ec6fdc1c3ba04f7a56db32c5ff5ff38962dc4cd0ffdef5beaa0ce2eb77e2"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 23922586,
         "digest": "sha256:b90c53a0b69244e37b3f8672579fc3dec13293eeb574fa0fdddf02da1e192fd6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 203,
         "digest": "sha256:11fa52a0fdc084d7fc3bbcb774389fd37b148ee98e7829cea4af189735acf848"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 216,
         "digest": "sha256:12d9697316a6aeb3945a585c945100693cfa06a7019062418eface3c8722c800"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 967332469,
         "digest": "sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 69843,
         "digest": "sha256:5d7549be9defb9c846d0b0bdf419adf7b917f261f82ced6d746d0b6786503e24"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 643,
         "digest": "sha256:6557d2446dac29c7fca3113ca8690d5b17af292d07b9d8a7ccc92b3e15a2d9c2"
      }
   ]

And GET/HEAD calls for blobs

http --header GET https://quay.io/v2/pulp/pulp-fixtures/blobs/sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5
HTTP/1.1 302 FOUND
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Connection: close
Content-Length: 1225
Content-Type: text/html; charset=utf-8
Date: Tue, 26 May 2020 15:16:38 GMT
Docker-Content-Digest: sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5
Location: https://cdn.quay.io/sha256/55/55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5?Expires=1590506798&Signature=LSMOGnm54aneTVgXbWfuPJEhFSDJKBdHgRs8cubnwChsi~9C6Je-wI18thPlb6j3d2RrVzcsJiDSweTSa~Ni2MJ2MfqnujDf~lT9dfBSMHjmfTPdVhql6YvImd9pL5CvRuxD6dvFIpmA2Rs0H-Mwi9w6ud7TWdZZ5trkoA-vvrQtK-GQ95TktgGhdUszqUyo6wSQHrgW1WISeOveddHgrKmbYJDNX~rQBLVjNZYFKBoyO-oHTgmgNID73xTXdpHu54VqK0A9FVsaCdRsafygU52W4QhknRM7Pr77IwbR9zPNWCHDw7pnr22NR7aVhtpE9Tma4Q5VrTSsjb0ZO9ra8w__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA
Server: nginx/1.12.1
Strict-Transport-Security: max-age=63072000; preload
X-Frame-Options: DENY

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$  http --header HEAD https://quay.io/v2/pulp/pulp-fixtures/blobs/sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=31436000
Connection: close
Content-Length: 967332469
Content-Type: application/octet-stream
Date: Tue, 26 May 2020 15:16:50 GMT
Docker-Content-Digest: sha256:55f96fcd2154dea0366210d95996aa7f17c3a1d5a8ff86da075c038b56fe17b5
Server: nginx/1.12.1
Strict-Transport-Security: max-age=63072000; preload
X-Frame-Options: DENY

Actions #5

Updated by mdellweg almost 4 years ago

You just reminded me about what i really wanted to try. If i "HEAD" the blob instead of "GET" i also do not get the redirect. Can you verify that for the Manifests?

Actions #6

Updated by ipanova@redhat.com almost 4 years ago

i believe it is enough to have set docker-content-digest for manifest only and not for blobs

This is dockerhub and headers it returns for blobs

$ ./docker-token library/busybox@sha256:d9cbbca60e5f0fc028b13c01e70e96cc2eefe7858ccc5a33c9a8faa8b03286aa
{'Date': 'Tue, 26 May 2020 12:39:13 GMT', 'Content-Type': 'application/octet-stream', 'Content-Length': '760847', 'Connection': 'keep-alive', 'Set-Cookie': '__cfduid=de5de30960a2908ef16c99893331488441590496753; expires=Thu, 25-Jun-20 12:39:13 GMT; path=/; domain=.production.cloudflare.docker.com; HttpOnly; SameSite=Lax; Secure', 'CF-Ray': '59978ec36c57f9e2-PRG', 'Accept-Ranges': 'bytes', 'Age': '1093147', 'Cache-Control': 'public, max-age=14400', 'ETag': '"17d26c5e6d0b638492f33daafaeb7d3b"', 'Expires': 'Tue, 26 May 2020 16:39:13 GMT', 'Last-Modified': 'Wed, 13 May 2020 20:48:14 GMT', 'Vary': 'Accept-Encoding', 'CF-Cache-Status': 'HIT', 'cf-request-id': '02f2978e210000f9e2d2378200000001', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'x-amz-id-2': 'TcdIEav8uoiC/qt6t6DXRWJZuy6KFhOUy9e5VedCjLn2JvXNYLFa5tL4sjTLTeHfw++fT8zmrWs=', 'x-amz-request-id': 'CA2C4EE257F33D9E', 'x-amz-version-id': 'NINdcIJ9IDpfyrDrWtGoiEcTjqEnomF1', 'Server': 'cloudflare'}
Actions #7

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 73 to Sprint 74
Actions #8

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 74 to Sprint 75
Actions #9

Updated by ipanova@redhat.com almost 4 years ago

  • Sprint/Milestone set to 2.0.0
Actions #10

Updated by ipanova@redhat.com almost 4 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #11

Updated by mdellweg over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF