Pulp

It seems to be a low priority on the aiohttp list to complete https://github.com/aio-libs/aiohttp/issues/845#issuecomment-313141691. But reading the whole thread, it seems that they are not against a contribution, and the main problem is a test setup.

I setup squid on a dev box and configured it to also support https_port like this:

1. Install with sudo dnf install squid -y
2. Make certificates with these commands (and use your hostname):

sudo openssl genrsa -out /etc/ssl/key.pem 4096
sudo openssl req -new -x509 -key /etc/ssl/key.pem -out /etc/ssl/cert.pem -days 1826

3. Add this line to the top of your /etc/squid/squid.conf: https_port 3130 tls-cert=/etc/ssl/cert.pem tls-key=/etc/ssl/key.pem

Then do some testing with curl.

Make a no proxy request: curl -v http://example.com Make a request going through the http-based proxy: curl -v --proxy https://localhost:3128 http://example.com Make a request going through the https proxy. curl -v --proxy-insecure --proxy https://localhost:3130 http://example.com

You have to use the --proxy-insecure because the certificate squid is using is self-signed. I couldn't easily get a request to https://example.com working, but that's not important for this bugfix because the proxy itself is TLS enabled.

Then have pulp_file sync through the http based proxy with:

#!/bin/bash

set -ev

repo_name="repo$RANDOM"
remote_name="remote$RANDOM"
distribution_name="distribution$RANDOM"
base_path="$RANDOM"
url="http://fixtures.pulpproject.org/file/PULP_MANIFEST"
proxy_url="http://pulp3-source-fedora33.fedora.example.com:3128/"

pulp file remote create --name $remote_name --url $url --proxy-url $proxy_url
pulp file repository create --name $repo_name --remote $remote_name
pulp file repository sync --name $repo_name
pulp file publication create --repository $repo_name
pulp file distribution create --base-path $base_path --repository $repo_name --name $distribution_name

echo "Created, synced, published repo $repo_name at distribution base path '$base_path'"

Then reproduce the issue by having it sync through the HTTPS proxy:

#!/bin/bash

set -ev

repo_name="repo$RANDOM"
remote_name="remote$RANDOM"
distribution_name="distribution$RANDOM"
base_path="$RANDOM"
url="http://fixtures.pulpproject.org/file/PULP_MANIFEST"
proxy_url="https://pulp3-source-fedora33.fedora.example.com:3130/"

pulp file remote create --name $remote_name --url $url --proxy-url $proxy_url
pulp file repository create --name $repo_name --remote $remote_name
pulp file repository sync --name $repo_name
pulp file publication create --repository $repo_name
pulp file distribution create --base-path $base_path --repository $repo_name --name $distribution_name

echo "Created, synced, published repo $repo_name at distribution base path '$base_path'"

It should show you: Error: Task /pulp/api/v3/tasks/ba5ecbf2-f7a8-48db-89e3-2c50af8fb8a6/ failed: 'Only http proxies are supported'

Here's an update of the key points I've learned so far.

• I've been mostly discussing this on this aiohttp PR https://github.com/aio-libs/aiohttp/pull/5992. Lots of details there
• aiohttp doesn't do TLS in TLS correctly because Python doesn't do it correctly https://bugs.python.org/issue37179. This is close to being resolved and will be fixed in either 3.10 (maybe?) or 3.11 (likely).
• Until this bug is fixed aiohttp and other web frameworks would have to carry a workaround. Other ones like httpx do carry this.

So there are two options I see:

1. Continue to try to fix aiohttp's TLS handshake workaround code
2. Accept this isn't working until aiohttp fixes it properly after Python fixes asyncio

Here's another way to setup the proxy using a python-based proxy proxy.py

# Make your keys
sudo openssl genrsa -out /etc/ssl/key.pem 4096
sudo openssl req -new -x509 -key /etc/ssl/key.pem -out /etc/ssl/cert.pem -days 1826

# Install proxy and run it
pip install proxy
proxy --hostname 0.0.0.0 --cert-file /etc/ssl/cert.pem --key-file /etc/ssl/key.pem

# Test it with curl in another tty
curl -v --proxy-insecure --proxy https://127.0.0.1:8899 https://example.com