Issue #9595
closedHEAD requests on the artefacts from S3 storage recieve 403
Description
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http https://pulp3-source-fedora34.fluffy.example.com/pulp/api/v3/distributions/file/file/9034d885-babb-4d19-81b3-091bd9f63a7f/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 398
Content-Type: application/json
Correlation-ID: c5d10e5eaf7c40518b04130e3a2b22d3
Date: Wed, 01 Dec 2021 12:41:05 GMT
Referrer-Policy: same-origin
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
{
"base_path": "vewtf",
"base_url": "https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/",
"content_guard": null,
"name": "cizwr",
"publication": "/pulp/api/v3/publications/file/file/10c34224-83ff-40b5-a47e-1453a13cbc88/",
"pulp_created": "2021-12-01T12:34:22.111939Z",
"pulp_href": "/pulp/api/v3/distributions/file/file/9034d885-babb-4d19-81b3-091bd9f63a7f/",
"pulp_labels": {},
"repository": null
}
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 343
Content-Type: text/html; charset=utf-8
Date: Wed, 01 Dec 2021 12:41:14 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
X-PULP-CACHE: HIT
<!DOCTYPE html>
<html>
<body>
<ul>
<li><a href="PULP_MANIFEST">PULP_MANIFEST</a></li>
<li><a href="test_upload.txt">test_upload.txt</a></li>
</ul>
</body>
</html>
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow HEAD https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/test_upload.txt
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Date: Wed, 01 Dec 2021 12:41:20 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Accept-Encoding
X-Amz-Request-Id: 16BCA1FED0CD6B9E
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/test_upload.txt
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment;filename=test_upload.txt
Content-Length: 11
Content-Security-Policy: block-all-mixed-content
Content-Type: text/plain
Date: Wed, 01 Dec 2021 12:41:28 GMT
ETag: "eef16594e73fc257de8125c7f1727a95"
Last-Modified: Wed, 01 Dec 2021 12:34:12 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Request-Id: 16BCA200B53E2618
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
rzwdbspfbe
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$
Presigned URLs allow one type of HTTP request method, which is defined at their creation time. By default, Boto3 creates presigned URLs that permit only the HTTP GET method however, the request method can be specified. This is exposed in django-storages
S3Boto3Storage.url
as http_method.
Related issues
Updated by ipanova@redhat.com about 3 years ago
- Copied from Issue #9586: Container clients fail to retrieve artefacts from S3 storage added
Updated by ipanova@redhat.com about 3 years ago
- Description updated (diff)
The solution is to provide the httd_method
Updated by ipanova@redhat.com about 3 years ago
- Subject changed from HEAD requests on the artefacts from S3 storage reciee 403 to HEAD requests on the artefacts from S3 storage recieve 403
Updated by ipanova@redhat.com about 3 years ago
Passing in the httd_method does not seem to help because of enabled redis caching.
I see this weird behavior ( without any code modification). The first request whether it was called with HEAD or GET passes, but the following fails:
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 343
Content-Type: text/html
Date: Wed, 01 Dec 2021 13:14:34 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
X-PULP-CACHE: MISS
<!DOCTYPE html>
<html>
<body>
<ul>
<li><a href="PULP_MANIFEST">PULP_MANIFEST</a></li>
<li><a href="test_upload.txt">test_upload.txt</a></li>
</ul>
</body>
</html>
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow HEAD https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/test_upload.txt
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment;filename=test_upload.txt
Content-Length: 11
Content-Security-Policy: block-all-mixed-content
Content-Type: text/plain
Date: Wed, 01 Dec 2021 13:14:50 GMT
ETag: "1648d5c8ed2c653ae7d454c4476ace69"
Last-Modified: Wed, 01 Dec 2021 13:13:42 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Accept-Encoding
X-Amz-Request-Id: 16BCA3D2CF1F907D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/test_upload.txt
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 529
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Wed, 01 Dec 2021 13:14:58 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Request-Id: 16BCA3D4A0E8FA6D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Key>artifact/47/8a9c72939d4c892d284f11b12698d3f9dbd6a20814ce71ea208c21eff464a9</Key>
<BucketName>pulp3</BucketName>
<Resource>/pulp3/artifact/47/8a9c72939d4c892d284f11b12698d3f9dbd6a20814ce71ea208c21eff464a9</Resource>
<RequestId>16BCA3D4A0E8FA6D</RequestId>
<HostId>55f50e2c-3c81-4e96-a864-b27191fbc12c</HostId>
</Error>
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/PULP_MANIFEST
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment;filename=PULP_MANIFEST
Content-Length: 84
Content-Security-Policy: block-all-mixed-content
Content-Type: application/octet-stream
Date: Wed, 01 Dec 2021 13:15:32 GMT
ETag: "024544ef7c93d71c2184b0eb9ac31de0"
Last-Modified: Wed, 01 Dec 2021 13:13:48 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Request-Id: 16BCA3DCB8F3A28D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
test_upload.txt,478a9c72939d4c892d284f11b12698d3f9dbd6a20814ce71ea208c21eff464a9,11
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow HEAD https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/PULP_MANIFEST
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Date: Wed, 01 Dec 2021 13:15:38 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Accept-Encoding
X-Amz-Request-Id: 16BCA3DE14AAC780
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
(pulp) [vagrant@pulp3-source-fedora34 _scripts]$
Updated by ipanova@redhat.com about 3 years ago
per IRC convo there are changes that need to be done: 1 - change the keys for object storage to use the request method 2 - pass the request method to the url generation like the contributor did in pulp_container
Updated by ipanova@redhat.com about 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ipanova@redhat.com
Updated by pulpbot about 3 years ago
- Status changed from ASSIGNED to POST
Added by ipanova@redhat.com almost 3 years ago
Updated by ipanova@redhat.com almost 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulpcore|c79da9cbb5dc2637eb3331e72723f933dbff80f5.
Updated by pulpbot almost 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Fixed 403 on artifact retrieval from S3 when caching is enabled.
closes #9595