Story #4040
Updated by bmbouter over 5 years ago
h2. Problem The ArtifactDownloader has a max_downloads restrictor. This is an identical feature to the connection limiting offered with aiohttp itself. Specifically aiohttp offers two features with its TCPConnector: From "their docs":https://docs.aiohttp.org/en/stable/client_reference.html#tcpconnector * limit (int) – total number simultaneous connections. If limit is None the connector has no limit (default: 100). * limit_per_host (int) – limit simultaneous connections to the same endpoint. Endpoints are the same if they are have equal (host, port, is_ssl) triple. If limit is 0 the connector has no limit (default: 0). h2. Is this really a duplication? Yes, it is verified via wireshark analysis. A max_downloader=20 and a limit=20 for the same number of TCP connections to the server. h2. Solution Add connection_limit and connection_limit_per_host to the Remote like the rest of the attributes, e.g. proxy settings. connection_limit in Pulp maps to TCPConnector.limit in aiohttp connection_limit_per_host in Pulp maps to TCPConnector.limit_per_host in aiohttp h2. What are the right default to Pulp Like Pulp2, we should default connection_limit=5. We should default connection_limit_per_host to 0 so it's disabled by default. If unset on a remote, a user will receive these default values. h2. How to Implement This configuration should be read from the Remote and applied to the session by the Factory just like how the other settings are done. Since an unset value should provide default even if unset, the default actually like in the Factory code. That is "here":https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/download/factory.py#L62-L103
.