Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #9595

closed

HEAD requests on the artefacts from S3 storage recieve 403

Added by ipanova@redhat.com about 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
ipanova@redhat.com
Category:
-
Sprint/Milestone:
3.17.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 111
Quarter:

Description

(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http  https://pulp3-source-fedora34.fluffy.example.com/pulp/api/v3/distributions/file/file/9034d885-babb-4d19-81b3-091bd9f63a7f/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 398
Content-Type: application/json
Correlation-ID: c5d10e5eaf7c40518b04130e3a2b22d3
Date: Wed, 01 Dec 2021 12:41:05 GMT
Referrer-Policy: same-origin
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "base_path": "vewtf",
    "base_url": "https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/",
    "content_guard": null,
    "name": "cizwr",
    "publication": "/pulp/api/v3/publications/file/file/10c34224-83ff-40b5-a47e-1453a13cbc88/",
    "pulp_created": "2021-12-01T12:34:22.111939Z",
    "pulp_href": "/pulp/api/v3/distributions/file/file/9034d885-babb-4d19-81b3-091bd9f63a7f/",
    "pulp_labels": {},
    "repository": null
}


(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 343
Content-Type: text/html; charset=utf-8
Date: Wed, 01 Dec 2021 12:41:14 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
X-PULP-CACHE: HIT

        <!DOCTYPE html>
        <html>
            <body>
                <ul>
                
                    <li><a href="PULP_MANIFEST">PULP_MANIFEST</a></li>
                
                    <li><a href="test_upload.txt">test_upload.txt</a></li>
                
                </ul>
            </body>
        </html>
        


(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow HEAD https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/test_upload.txt
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Date: Wed, 01 Dec 2021 12:41:20 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Accept-Encoding
X-Amz-Request-Id: 16BCA1FED0CD6B9E
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block



(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/vewtf/test_upload.txt
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment;filename=test_upload.txt
Content-Length: 11
Content-Security-Policy: block-all-mixed-content
Content-Type: text/plain
Date: Wed, 01 Dec 2021 12:41:28 GMT
ETag: "eef16594e73fc257de8125c7f1727a95"
Last-Modified: Wed, 01 Dec 2021 12:34:12 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Request-Id: 16BCA200B53E2618
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

rzwdbspfbe


(pulp) [vagrant@pulp3-source-fedora34 _scripts]$

Presigned URLs allow one type of HTTP request method, which is defined at their creation time. By default, Boto3 creates presigned URLs that permit only the HTTP GET method however, the request method can be specified. This is exposed in django-storages S3Boto3Storage.url as http_method.

Related issues

Copied from Container Support - Issue #9586: Container clients fail to retrieve artefacts from S3 storageCLOSED - CURRENTRELEASEActions
Actions
Copy link
 #1

Updated by ipanova@redhat.com about 3 years ago

  • Copied from Issue #9586: Container clients fail to retrieve artefacts from S3 storage added
Actions
Copy link
 #2

Updated by ipanova@redhat.com about 3 years ago

  • Description updated (diff)

The solution is to provide the httd_method

Actions
Copy link
 #3

Updated by ipanova@redhat.com about 3 years ago

  • Subject changed from HEAD requests on the artefacts from S3 storage reciee 403 to HEAD requests on the artefacts from S3 storage recieve 403
Actions
Copy link
 #4

Updated by ipanova@redhat.com about 3 years ago

Passing in the httd_method does not seem to help because of enabled redis caching.

I see this weird behavior ( without any code modification). The first request whether it was called with HEAD or GET passes, but the following fails:

(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 343
Content-Type: text/html
Date: Wed, 01 Dec 2021 13:14:34 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
X-PULP-CACHE: MISS

        <!DOCTYPE html>
        <html>
            <body>
                <ul>
                
                    <li><a href="PULP_MANIFEST">PULP_MANIFEST</a></li>
                
                    <li><a href="test_upload.txt">test_upload.txt</a></li>
                
                </ul>
            </body>
        </html>
        


(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow HEAD https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/test_upload.txt
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment;filename=test_upload.txt
Content-Length: 11
Content-Security-Policy: block-all-mixed-content
Content-Type: text/plain
Date: Wed, 01 Dec 2021 13:14:50 GMT
ETag: "1648d5c8ed2c653ae7d454c4476ace69"
Last-Modified: Wed, 01 Dec 2021 13:13:42 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Accept-Encoding
X-Amz-Request-Id: 16BCA3D2CF1F907D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block




(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/test_upload.txt
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 529
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Wed, 01 Dec 2021 13:14:58 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Request-Id: 16BCA3D4A0E8FA6D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <Key>artifact/47/8a9c72939d4c892d284f11b12698d3f9dbd6a20814ce71ea208c21eff464a9</Key>
  <BucketName>pulp3</BucketName>
  <Resource>/pulp3/artifact/47/8a9c72939d4c892d284f11b12698d3f9dbd6a20814ce71ea208c21eff464a9</Resource>
  <RequestId>16BCA3D4A0E8FA6D</RequestId>
  <HostId>55f50e2c-3c81-4e96-a864-b27191fbc12c</HostId>
</Error>


(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow GET https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/PULP_MANIFEST
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment;filename=PULP_MANIFEST
Content-Length: 84
Content-Security-Policy: block-all-mixed-content
Content-Type: application/octet-stream
Date: Wed, 01 Dec 2021 13:15:32 GMT
ETag: "024544ef7c93d71c2184b0eb9ac31de0"
Last-Modified: Wed, 01 Dec 2021 13:13:48 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Request-Id: 16BCA3DCB8F3A28D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

test_upload.txt,478a9c72939d4c892d284f11b12698d3f9dbd6a20814ce71ea208c21eff464a9,11


(pulp) [vagrant@pulp3-source-fedora34 _scripts]$ http --follow HEAD https://pulp3-source-fedora34.fluffy.example.com/pulp/content/krfad/PULP_MANIFEST
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Date: Wed, 01 Dec 2021 13:15:38 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Accept-Encoding
X-Amz-Request-Id: 16BCA3DE14AAC780
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block



(pulp) [vagrant@pulp3-source-fedora34 _scripts]$
Actions
Copy link
 #5

Updated by ipanova@redhat.com about 3 years ago

per IRC convo there are changes that need to be done: 1 - change the keys for object storage to use the request method 2 - pass the request method to the url generation like the contributor did in pulp_container

Actions
Copy link
 #6

Updated by ipanova@redhat.com about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions
Copy link
 #7

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST

Added by ipanova@redhat.com about 3 years ago

Revision c79da9cb | View on GitHub

Fixed 403 on artifact retrieval from S3 when caching is enabled.

closes #9595

Actions
Copy link
 #8

Updated by ipanova@redhat.com about 3 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #9

Updated by ipanova@redhat.com about 3 years ago

  • Sprint set to Sprint 111
Actions
Copy link
 #10

Updated by pulpbot about 3 years ago

  • Sprint/Milestone set to 3.17.0
Actions
Copy link
 #11

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF