Container Support

Explanation¶

OS: CentOS 8 pulpcore version: 3.16 pulp_container version: 2.9.0

Container clients (podman and docker) fail to retrieve container artefacts when using the S3 storage backend (Ceph RADOS gateway S3 API).

When a container client tries to access a container artefact, it makes HTTP HEAD requests successfully to the pulp/pulp_container API, which correctly issues HTTP 302 responses, redirecting clients to the presigned S3 URL of the container artefact. When the container client follows the redirect and sends a HTTP HEAD request to the presigned URL, it receives a HTTP 403 from the S3 API, which results in the container client failing to push/pull container artefacts.

Manually issuing a HTTP GET request directly to the presigned URL is successful. Also, issuing a HTTP GET request to the pulp/pulp_container API, which is redirected to the S3 presigned URL, is successful. The issue appears to be that the presigned URL does not permit HEAD requests sent by the container client that are redirected from the pulp/pulp_container API.

Presigned URLs allow one type of HTTP request method, which is defined at their creation time. By default, Boto3 creates presigned URLs that permit only the HTTP GET method however, the request method can be specified. This is exposed in django-storages S3Boto3Storage.url as http_method.

When generating a presigned URL, setting http_method to the same value as the method from the original request to the pulp/pulp_container API solves this issue. I hotfixed our pulp_container install to add this, and it seems to resolve the issue.

Steps to reproduce¶

Docker push¶

Client (receives HTTP 403 from S3 API):

(venv) [matta@matt-top ~]$ sudo docker push pulp.local/alpine/alpine:latest
The push refers to repository [pulp.local/alpine/alpine]
1a058d5342cc: Pushing [==================================================>]  5.886MB
error parsing HTTP 403 response body: unexpected end of JSON input: ""

Server (no obvious errors or issues):

Nov 25 11:21:34 pulp gunicorn[287340]: pulp [f234f7ac6a3849808ccd0ce3e533a018]: 127.0.0.1 - - [25/Nov/2021:11:21:34 +0000] "HEAD /v2/alpine/alpine/blobs/sha256:97518928ae5f3d52d4164b314a7e73654eb6>
Nov 25 11:21:35 pulp gunicorn[287340]: pulp [4323085440214bc7bfeca7baf6887565]: 127.0.0.1 - - [25/Nov/2021:11:21:35 +0000] "POST /v2/alpine/alpine/blobs/uploads/ HTTP/1.0" 202 0 "-" "docker/20.10.>
Nov 25 11:21:36 pulp gunicorn[287340]: pulp [e499fb41934d4c2fac4f3159fb64f79f]:  127.0.0.1 - - [25/Nov/2021:11:21:36 +0000] "PATCH /v2/alpine/alpine/blobs/uploads/d7d32fb0-69bb-4463-95aa-8eea37ba37>
Nov 25 11:21:38 pulp gunicorn[287340]: pulp [f1bca6f1d1b04bde8a7094012d6c3b27]:  127.0.0.1 - - [25/Nov/2021:11:21:38 +0000] "PUT /v2/alpine/alpine/blobs/uploads/d7d32fb0-69bb-4463-95aa-8eea37ba37ac>
Nov 25 11:21:38 pulp gunicorn[287340]: pulp [90fef9af936b4a74b59ad8cfb71e35b9]: 127.0.0.1 - - [25/Nov/2021:11:21:38 +0000] "HEAD /v2/alpine/alpine/blobs/sha256:97518928ae5f3d52d4164b314a7e73654eb6>

HTTP HEAD to pulp API following redirect (BAD)¶

(venv) [matta@matt-top ~]$ http --follow --auth-type=jwt --auth="SNIP" HEAD /v2/alpine/alpine/blobs/sha256:97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 191
Content-Type: application/xml
Date: Thu, 25 Nov 2021 11:31:57 GMT
x-amz-request-id: tx000000000000001d60235-00619f742d-686a2d8-default

HTTP HEAD to pulp API without following redirect (GOOD)¶

(venv) [matta@matt-top ~]$ http --auth-type=jwt --auth="SNIP"  HEAD /v2/alpine/alpine/blobs/sha256:97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
HTTP/1.1 302 Found
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 0
Content-Type: text/html; charset=utf-8
Correlation-ID: e81d9bfd66504c03aac18040c1385cd7
Date: Thu, 25 Nov 2021 11:27:33 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: https://correct.location/artifact/etc/etc
Referrer-Policy: same-origin
Server: nginx/1.14.1
Strict-Transport-Security: max-age=15768000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

HTTP GET to pulp API following redirect (GOOD)¶

(venv) [matta@matt-top ~]$ http --follow --auth-type=jwt --auth="SNIP" GET /v2/alpine/alpine/blobs/sha256:97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment; filename=518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
Content-Length: 2822981
Content-Type: application/vnd.docker.image.rootfs.diff.tar.gzip
Date: Thu, 25 Nov 2021 11:34:14 GMT
ETag: "c90c6064d27d5f3b1745980ded0e8dcf"
Last-Modified: Thu, 25 Nov 2021 11:21:37 GMT
x-amz-request-id: tx000000000000001c7d672-00619f74b6-6ac0692-default
x-rgw-object-type: Normal

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

HTTP HEAD to pulp API following redirect with http_method hotfix in place (GOOD)¶

(venv) [matta@matt-top ~]$ http --follow --auth-type=jwt --auth="SNIP" HEAD /v2/alpine/alpine/blobs/sha256:97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Disposition: attachment; filename=518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
Content-Length: 2822981
Content-Type: application/vnd.docker.image.rootfs.diff.tar.gzip
Date: Thu, 25 Nov 2021 11:42:21 GMT
ETag: "c90c6064d27d5f3b1745980ded0e8dcf"
Last-Modified: Thu, 25 Nov 2021 11:21:37 GMT
x-amz-request-id: tx000000000000001d613e3-00619f769d-686a2d8-default
x-rgw-object-type: Normal

Expected results¶

Container clients are able to interact with container artefacts stored on S3.

Actual results¶

Container clients receive HTTP 403 from S3 API when they try to access presigned URLs, which means that they are unable to interact with artefacts stored on S3.

Proposed fix¶

Add http_method argument to artifact.file.storage.url when redirecting to S3 URLs.