Project

Profile

Help

Issue #8875

pulp fails downloading packages with special symbols from Amazon Linux repositories

Added by vchepkov 9 days ago. Updated 6 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
Operator
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

While syncing Amazon Linux repository, pulp 2 (katello 3.15) and pulp 3 (katello 4) fail to download packages with special symbols, like "+":

Jun  6 11:57:44 foreman pulpcore-worker-1[19215]: aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/844a030e99d8e563fb9e83fa59b7c9dd76eea3f2a6e9ef71ed02e9420fec4f0c/libstdc++-7.2.1-2.amzn2.0.1.x86_64.rpm')

I have opened case with AWS and they told me that S3 backend requires special symbols to be encoded: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines-special-handling

Characters that might require special handling

The following characters in a key name might require additional code handling and likely need to be URL encoded or referenced as HEX. Some of these are non-printable characters that your browser might not handle, which also requires special handling:

Ampersand ("&")
Dollar ("$")
ASCII character ranges 00–1F hex (0–31 decimal) and 7F (127 decimal)
'At' symbol ("@")
Equals ("=")
Semicolon (";")
Colon (":")
Plus ("+")
Space – Significant sequences of spaces might be lost in some uses (especially multiple spaces)
Comma (",")
Question mark ("?")

How to reproduce: Add Amazon Core 2 repository: http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list

History

#2 Updated by ggainey 6 days ago

Fun.

The URL specified above does fail with a 403. Urlencoding the RPM name does work:

~/Downloads $ wget http://amazonlinux.us-east-1.amazonaws.com/blobstore/844a030e99d8e563fb9e83fa59b7c9dd76eea3f2a6e9ef71ed02e9420fec4f0c/libstdc%2B%2B-7.2.1-2.amzn2.0.1.x86_64.rpm
--2021-06-10 14:54:03--  http://amazonlinux.us-east-1.amazonaws.com/blobstore/844a030e99d8e563fb9e83fa59b7c9dd76eea3f2a6e9ef71ed02e9420fec4f0c/libstdc%2B%2B-7.2.1-2.amzn2.0.1.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 2600:1fa0:8041:4251:34d8:8c8e::, 52.217.105.46
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|2600:1fa0:8041:4251:34d8:8c8e::|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 447944 (437K) [binary/octet-stream]
Saving to: ‘libstdc++-7.2.1-2.amzn2.0.1.x86_64.rpm.1’

libstdc++-7.2.1-2.amzn2.0.1.x86_64.rpm.1                                        100%[====================================================================================================================================================================================================>] 437.45K  --.-KB/s    in 0.04s   

2021-06-10 14:54:03 (9.75 MB/s) - ‘libstdc++-7.2.1-2.amzn2.0.1.x86_64.rpm.1’ saved [447944/447944]

~/Downloads $ 

No other RPM repository has this issue (Fedora, RHEL, CentOS, SUSE) - altho a quick test suggests that Fedora, at least, would continue to work in the face of urlencoded-rpm-names.

Please register to edit this issue

Also available in: Atom PDF