https://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-12-04T16:43:17ZPulpContainer Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=653042020-12-04T16:43:17Zjsherril@redhat.comjsherril@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/65304/diff?detail_id=65423">diff</a>)</li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=653052020-12-04T16:47:23Zjsherril@redhat.comjsherril@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/65305/diff?detail_id=65424">diff</a>)</li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=653062020-12-04T16:57:04Zjsherril@redhat.comjsherril@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Tags</strong> <i>Katello</i> added</li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=653472020-12-04T18:16:25Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>The digest changes every time because you are requesting shema1 ( because i do not see any accept_headers in you request)
Pulp3 converts manifest <code>latest</code> which is probably schema2 into schema1.
Conversion on the fly creates new digest every time.</p>
<p>Can you provide more info and logs/tracebacks how pulp2 fails when syncing from pulp3?</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=653522020-12-04T19:38:03Zdkliban@redhat.com
<ul></ul><p>Here is a theory that I need to test out:</p>
<p>The initial request is sent with the correct headers. However, when pulp 3 sends a redirect to the content app, nectar fails to send those headers when following the redirect.</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=653992020-12-07T15:24:35Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li><li><strong>Sprint</strong> set to <i>Sprint 87</i></li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=655002020-12-07T22:58:50Zjsherril@redhat.comjsherril@redhat.com
<ul></ul><p>For blobs, it looks to me like they are coming in with</p>
<pre><code>ACCEPT= */*
</code></pre>
<p>Strangely when we fix our installer to put back katello as being in front of '/v2/' requests the problem goes away, even though we are forwarding the Accept header to the api app (and then redirecting the client to the content app directly). I don't see why that would make a difference since the request to the content app comes straight from the client directly</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=655042020-12-08T13:32:20Zjsherril@redhat.comjsherril@redhat.com
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li></ul><p>This was on pulp-container 2.1.0</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=657702020-12-15T16:17:11Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Sprint</strong> deleted (<del><i>Sprint 87</i></del>)</li></ul><p>Taking off the sprint for now, will come back to it later</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=658862020-12-21T11:27:54Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>I had an enlightenment on where might be the issue and how to reproduce. Instead of using pulp2migration box which has some sever misconfiguration that prevented from proper testing I used 2 boxes - pulp3-fedora box and pulp2 migration box.</p>
<pre><code>$ pulp-admin docker repo sync run --repo-id pulp3-repo
+----------------------------------------------------------------------+
Synchronizing Repository [pulp3-repo]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Task Failed
The Manifest digest does not match the expected value. The remote feed announced
a digest of
sha256:6e11c15668e7d20d60fe5a790c16b0aedc90b725c7f715aeef8ccc7e22fb7ee6, but the
downloaded digest was
sha256:94c54600f6939911c4ed74fae49c78e00808544803a4a883e65d697a7e89c4d3.
</code></pre>
<p>The digest does not match because in the headers we are sending the digest of the non-converted manifest. This is not correct. The digest should contain <strong>converted schema payload with the stripped out signature</strong></p>
<p>The changes will happen in the schema_conversion code.
<a href="https://github.com/pulp/pulp_container/blob/master/pulp_container/app/schema_convert.py#L35" class="external">https://github.com/pulp/pulp_container/blob/master/pulp_container/app/schema_convert.py#L35</a>
<a href="https://github.com/pulp/pulp_container/blob/master/pulp_container/app/schema_convert.py#L43" class="external">https://github.com/pulp/pulp_container/blob/master/pulp_container/app/schema_convert.py#L43</a>
I suggest the <code>Schema2toSchema1Converter.convert</code> method return signed manifest as well as digest of the manifest without the signature <a href="https://github.com/pulp/pulp_container/blob/master/pulp_container/app/schema_convert.py#L97" class="external">https://github.com/pulp/pulp_container/blob/master/pulp_container/app/schema_convert.py#L97</a></p>
<p>I made some calls to dockerhub and here is the proof that the digest sent in the header is the digest calculated on manifest without the signature. The digest does not change, if it would contain the signature it would change with every call since signature has different fingerprint in every call.</p>
<pre><code>(call1)
$ ./docker-token library/busybox:latest
{'Content-Length': '2735', 'Content-Type': 'application/vnd.docker.distribution.manifest.v1+prettyjws', 'Docker-Content-Digest': 'sha256:af39243ae92c12504f260709da43f1b4bd17a802a86a367ffcd7f4913688d92a', 'Docker-Distribution-Api-Version': 'registry/2.0', 'Etag': '"sha256:af39243ae92c12504f260709da43f1b4bd17a802a86a367ffcd7f4913688d92a"', 'Date': 'Mon, 21 Dec 2020 10:52:14 GMT', 'Strict-Transport-Security': 'max-age=31536000', 'RateLimit-Limit': '200;w=21600', 'RateLimit-Remaining': '199;w=21600'}
(call2)
[ipanova@fluffy pulp_container]$ ./docker-token library/busybox:latest
{'Content-Length': '2735', 'Content-Type': 'application/vnd.docker.distribution.manifest.v1+prettyjws', 'Docker-Content-Digest': 'sha256:af39243ae92c12504f260709da43f1b4bd17a802a86a367ffcd7f4913688d92a', 'Docker-Distribution-Api-Version': 'registry/2.0', 'Etag': '"sha256:af39243ae92c12504f260709da43f1b4bd17a802a86a367ffcd7f4913688d92a"', 'Date': 'Mon, 21 Dec 2020 10:52:21 GMT', 'Strict-Transport-Security': 'max-age=31536000', 'RateLimit-Limit': '200;w=21600', 'RateLimit-Remaining': '199;w=21600'}
</code></pre>
<p>versus we are sending digest of the original manifest</p>
<pre><code>(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 _scripts]$ http HEAD "http://pulp3-source-fedora32.fluffy.example.com/v2/test/manifests/manifest_e" --follow
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PUT, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 942
Correlation-ID: 275fbd944a0c421e9f9e8d114eff74aa
Date: Mon, 21 Dec 2020 11:14:01 GMT
Docker-Content-Digest: sha256:e7300fcf2becf0e60628ee003902f9e4b70b3ea1782f766fd5d45b59a2126f50
Docker-Distribution-Api-Version: registry/2.0
Location: /v2/test/manifests/sha256:e7300fcf2becf0e60628ee003902f9e4b70b3ea1782f766fd5d45b59a2126f50
Server: nginx/1.18.0
X-Frame-Options: SAMEORIGIN
$ http GET ":24817/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container/78452742-15e0-4580-9ae1-da7e67232d32/versions/1/&name=manifest_e"
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 305
Content-Type: application/json
Correlation-ID: 59c853adcd8148738658dc630d047ccc
Date: Mon, 21 Dec 2020 11:25:02 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"name": "manifest_e",
"pulp_created": "2020-12-21T10:41:35.656901Z",
"pulp_href": "/pulp/api/v3/content/container/tags/a2b6e5b8-ba4d-4d6f-89f3-a0415dc12c57/",
"tagged_manifest": "/pulp/api/v3/content/container/manifests/d57301ab-0abb-4feb-a032-66ec930e7f84/"
}
]
}
(pulp) [vagrant@pulp3-source-fedora32 _scripts]$ http GET :24817/pulp/api/v3/content/container/manifests/d57301ab-0abb-4feb-a032-66ec930e7f84/?fields=digest
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 84
Content-Type: application/json
Correlation-ID: a3ba96b2f8d04edf970125deee394ea6
Date: Mon, 21 Dec 2020 11:25:24 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"digest": "sha256:e7300fcf2becf0e60628ee003902f9e4b70b3ea1782f766fd5d45b59a2126f50"
}
</code></pre> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=658872020-12-21T11:40:28Zmdellweg
<ul></ul><p>So this digest is nothing we can know in a head request without performing the actual conversion? That means, as the clients usually perform a head first, we would need to convert things twice regularly. Or we need to turn them into alternate artifacts in the db.</p>
<p>When you do the request to dockerhub twice, do the blobs you receive change? Maybe <em>they</em> store the converted result.</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=658882020-12-21T12:44:03Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>mdellweg wrote:</p>
<blockquote>
<p>So this digest is nothing we can know in a head request without performing the actual conversion? That means, as the clients usually perform a head first, we would need to convert things twice regularly. Or we need to turn them into alternate artifacts in the db.</p>
<p>When you do the request to dockerhub twice, do the blobs you receive change? Maybe <em>they</em> store the converted result.</p>
</blockquote>
<p>Blobs do not change, the only object that can change is the manifest converted on the fly.
HEAD request is usually made to check on an existing resource, since the manifest gets converted on the fly it is questionable to consider it as an existing resource. I suggest to issue 404 if there is no such tag that matches the list of accepted_headers sent along with the request.
This would be the place to adjust the logic in case of us deciding to return digest of converted schema <a href="https://github.com/pulp/pulp_container/blob/master/pulp_container/app/registry_api.py#L545" class="external">https://github.com/pulp/pulp_container/blob/master/pulp_container/app/registry_api.py#L545</a> dockerhub returns a 400 on such request</p>
<p>I also noticed that we are not handling properly tag redirects <a href="https://github.com/pulp/pulp_container/blob/master/pulp_container/app/redirects.py#L39" class="external">https://github.com/pulp/pulp_container/blob/master/pulp_container/app/redirects.py#L39</a>
We should also take into account the media_types as in S3 redirects.</p>
<pre><code>$ curl -X GET -H "Accept:application/vnd.docker.distribution.manifest.list.v2+json" "http://pulp3-source-fedora32.fluffy.example.com/v2/test3/manifests/ml_i" -L
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 735,
"digest": "sha256:94391db5d7dae06e2e463ca41a0b8b73381817d3ab23d7a52c16db60b89a966e",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 735,
"digest": "sha256:bdc42bf398edffb7d5cee329d16bae00439fcc7ee963e8089f293018268ffae1",
"platform": {
"architecture": "amd64",
"os": "linux"
}
}
]
}(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 _scripts]$ curl -X GET -H "Accept:application/vnd.docker.distribution.manifest.v1+json" "http://pulp3-source-fedora32.fluffy.example.com/v2/test3/manifests/ml_i" -L
500 Internal Server Error
</code></pre> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=658892020-12-21T12:53:13Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Sprint</strong> set to <i>Sprint 88</i></li></ul><p>Podman pull passes because in the first place it always sends all the media_types in the accept headers so no conversion logic is involved and the redirects happen to the original object.</p>
<p>We need to fix redirects(1) as well as schema conversion(2) to enable pulp3 to pulp3 sync as well as pulp3 to pulp2 sync.
(3) handle head request for tags, my suggestions would be to issue 404 if there is no such tag that matches the list of accepted_headers sent along with the request</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=660262021-01-07T09:33:29Zlmjachky
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>lmjachky</i></li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=660282021-01-07T12:57:00Zlmjachky
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>NEW</i></li><li><strong>Assignee</strong> deleted (<del><i>lmjachky</i></del>)</li></ul><p>I am unassigning myself because I cannot sync from docker.io due to the pull rate limit.</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=661162021-01-11T10:28:28Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>lmjachky</i></li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=667462021-01-22T20:32:35Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 88</i> to <i>Sprint 89</i></li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=667622021-01-23T13:30:47Zpulpbot
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p>PR: <a href="https://github.com/pulp/pulp_container/pull/209" class="external">https://github.com/pulp/pulp_container/pull/209</a></p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=668402021-01-26T10:12:18ZAnonymous
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="Return the digest of an unsigned manifest closes #7923" href="https://pulp.plan.io/projects/pulp_container/repository/68/revisions/0a362e236bcf2d5ecac15afd4d8d5d166f732637">0a362e236bcf2d5ecac15afd4d8d5d166f732637</a>.</p> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=673552021-02-08T16:46:47Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Sprint/Milestone</strong> set to <i>2.3.0</i></li></ul> Container Support - Issue #7923: manifest requests do not match advertised checksum under some situationshttps://pulp.plan.io/issues/7923?journal_id=673602021-02-08T17:42:22Zpulpbot
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul>