Actions
Issue #8875
closed
pulp fails downloading packages with special symbols from Amazon Linux repositories
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 102
Quarter:
Description
While syncing Amazon Linux repository, pulp 2 (katello 3.15) and pulp 3 (katello 4) fail to download packages with special symbols, like "+":
Jun 6 11:57:44 foreman pulpcore-worker-1[19215]: aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/844a030e99d8e563fb9e83fa59b7c9dd76eea3f2a6e9ef71ed02e9420fec4f0c/libstdc++-7.2.1-2.amzn2.0.1.x86_64.rpm')
I have opened case with AWS and they told me that S3 backend requires special symbols to be encoded: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines-special-handling
Characters that might require special handling
The following characters in a key name might require additional code handling and likely need to be URL encoded or referenced as HEX. Some of these are non-printable characters that your browser might not handle, which also requires special handling:
Ampersand ("&")
Dollar ("$")
ASCII character ranges 00–1F hex (0–31 decimal) and 7F (127 decimal)
'At' symbol ("@")
Equals ("=")
Semicolon (";")
Colon (":")
Plus ("+")
Space – Significant sequences of spaces might be lost in some uses (especially multiple spaces)
Comma (",")
Question mark ("?")
How to reproduce: Add Amazon Core 2 repository: http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
Related issues
Actions
.
Taught downloader to url-encode so odd backends can find content.
Amazon (for example) will reject RPMs with '+' in the filename if it's not url-encoded.
fixes #8875.