Issue #2677
closedOccasional HTTP 403 errors when downloading rpms from on_demand / lazy sync repositories
Description
The streamer is resulting in occasional 403s.
The patched version of the HTTPAdaptor included in the streamer package basically replicates the upstream patch[1] that has been accepted and merged to python-requests proposed/3.0.0. However, the patch to python-requests only addresses part of the problem. It propagates settings to the urllib3 PoolManager. However, the criteria (pool-key) used by the PoolManager to select a connection pool only includes (scheme, host, port). This means connections will be reused even when different certificates are specified. As of this PR[2], urllib3 was patched to have custom pool keys which (for ssl) includes the certificate path. Good effort. As a result, even when using the latest patched urllib3 (in F25) connections with different certificate paths will not reuse connections when different certificates are specified.
The latest patches to python-requests and urllib3 cannot be used as is with pulp2 because of how Pulp/Nectar handles certificates. Pulp stores certificates in the DB and passes to nectar as PEM strings. Nectar writes them to a temporary file and passes the path to requests. The certificate path will be different every time. This will cause the urllib3 PoolManager to create a new pool for every request which results in never sharing connections.
EL7 has urllib3 1.10
F25 has urllib3 1.15
[1] https://github.com/kennethreitz/requests/pull/3109
[2] https://github.com/shazow/urllib3/pull/830
Cache sessions in the streamer. closes #2677