Pulp

I was testing against dl.fedoraproject.org with various concurrency settings. A concurrency of 3 failed. ttereshc said on irc that a concurrency of 1 also failed...

I asked in #fedora-admin and they said
(1) fedora infra limits to 15 connections per ip. AIUI with asyncio and connection reuse we are using 1 connection so that shouldn't limit us.
(2) they looked the logs from my IP that received ServerDisconnectErrors and they showed all 200 OK. This is especially strange because if the server says all responses are 200 OK and my client disconnects something is wrong.

I wanted to test w/ other version of asyncio. I reproduced the same ServerDisconnect w/ the 3.3.2 and the latest 3.4.4.

I reproduced the issue against other non-fedora servers so I don't think this is specific to Fedora:

http://download1.rpmfusion.org/free/fedora/releases/28/Everything/x86_64/os/
http://download-ipv4.eng.brq.redhat.com/released/RHEL-7/7.5/Server/x86_64/os/

For the 503 errors I think we should remove giveup altogether and just have it try over and over. I think users want us to try at all costs basically.

Excellent findings. I'm curious about how Pulp 2 deals with these issues. Like how many concurrent tcp connections do we use by default in Pulp 2? And how does Pulp 2 handle 503 errors (ie does it retry or give up?)?

Pulp2 uses nectar for downloading, which defaults to 5 retries and 5 concurrent downloads. See the code here

RPM only overrides this if the user defines a value on the importer which is then overridden here

I'm not sure under which conditions the retry occurs.
I'm not sure how many tcp connections are used when connecting to a remote server.
I'm unclear at how 5 was determined as the "right" number.

Here's a summary I posted in IRC:

10:20 <mhrivnak> jortel bmbouter yes, years ago I setup an experiment that you can probably find detailed in the archives for one of the email lists. I can summarize it though.
10:21 <mhrivnak> I took a ton of RPMs from some linux distro and stuck them on a machine in phx2. That introduced real-world latency, but very consistent latency, because traffic was going over a dedicated connection from the office.
10:22 <mhrivnak> I tried a bunch of concurrency levels, and the max throughput was observed with 7 concurrent downloaders.
10:22 <mhrivnak> But the increase from about 4 to 7 was very slim.
10:23 <mhrivnak> So in the interest of being more "polite" to remote servers, we dialed 7 back to 5.
10:23 <mhrivnak> I think katello may have dialed that back to 4 even with the setting, if my memory is correct.
10:25 <mhrivnak> There have been a few occasions where users have observed that a remote server stopped responding to requests, or started responding with 4xx or 5xx codes, because they were receiving too many connections at once from pulp. In those cases, changing the setting to 2 or 3 has resolved it.

To the TCP question, each downloader establishes a single TCP connection between itself and any given remote server, and it reuses that until the remote server closes it (which may or may not happen). That's the ideal situation, because establishing new TCP connections can be surprisingly expensive, especially if you add TLS on top. Depending on the http client you use, you might need to tell it explicitly to re-use connections.

Worth knowing: HTTP 1.1 limits you to a single request/response at a time per connection. You cannot make concurrent requests over a single connection. One of the hot new features of HTTP 2.0 is that it does support concurrent requests over a single connection, but not a lot of clients and servers are using 2.0 yet. And that said, my expectation would be that N (where N > 1) concurrent downloads with http 2.0 over a single connection would be slower than using N connections. I would even suspect that N downloads over 1 connection with http 2.0 would be closer in performance to concurrency of 1 with http 1.1 than concurrency of N with http 1.1. It would be interesting to test, but I think the concurrent requests over a single connection are more useful for small request/response payloads where processing time is needed on the remote end. For example, loading a webpage that requires 25 requests for various small resources.

After doing some reading, here is some general info I found:

1. The aiohttp library is correctly reusing TCP connections. I verified this from wireshark data and it confirms the aiohttp claim in their docs.

2. HTTP 1.1 does support Pipelining , but aiohttp as a client does not support it AIUI. This issue in their tracker suggests they don't support it, and I can't find anything their client docs about it. I opened this to get more clarity on it https://github.com/aio-libs/aiohttp/issues/3310

Issue #4040 should resolve all of our issues except 2 cases:

1. Where server is responding w/ HTTP > 3XX error codes and Pulp isn't retrying correctly
2. Even with the settings from #4040 limiting the number of TCP connections, if the server decides to close all of them Pulp could still decide to reconnect. The @backoff behavior we haven't won't do this for us because we continually hand a closed connection to future downloaders even after the remote side closed it. This case is complicated enough that I'm moving it to its own ticket and also asking for input from upstream aiohttp.

So for the remainder of this issue I want to focus only on issue (1) that Pulp isn't retrying with >3XX HTTP status codes.

To fix that I recommend we:

Redo the giveup() method to retry anytime it receives an error raised from raise_for_status() in aiohttp. Currently we only retry for HTTP 429, so this would fix the problem that we're not trying for 503, and all the >3XX error codes.

I captured wireshark data for Pulp2, but since it works it's 800+ MB so I'm not going to post it here. I captured it with:

pulp-admin -u admin -p admin rpm repo create --repo-id rpmfusion --relative-url rpmfusion --feed http://download1.rpmfusion.org/free/fedora/releases/28/Everything/x86_64/os/
pulp-admin -u admin -p admin rpm repo sync run --repo-id rpmfusion

Here are a few quick observations from the wireshark data: Pulp2 is limiting to 5 concurrent connections and that is working for it. It's closing the TCP connections and recycling them more often. I see over 20 TCP connections made over the duration of the sync, but only 5 are connected at any given time. This is different from aiohttp at least which makes 5 connections immediately and never disconnects.

I'm going to continue my analysis.

I've been working on an easy way to reproduce this I made this client and server which will hold open N connections and then close the one that receives the SIGUSR1 signal. The idea is that a client shown closing all connections when one normally closes should show itself with a server like this.

server: https://gist.github.com/bmbouter/222b79bf453f8f9a77069e86e3b65ac8
client: https://gist.github.com/bmbouter/90c1eb9d0b476a0a674930aa33852b56

Run the server using gunicorn with something like:

gunicorn server:my_web_app --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 10

Then when you connect via the client you'll see which pids receive the connections. Send SIGUSR1 to one of the processes that shows it was connected to in the gunicorn output with kill -s SIGUSR1 <pid>

Strangely this does not show the issue. It shows the one connection closing with 200 OK, and the other remaining open.

I tried using pulp_file with a custom server able to act like a pulp_file repo, and it also did not reproduce. This suggests that it's not a bug in aiohttp. I went back and reproduced ServerDisconnectError against rpmfusion just to be sure the original problem still reproduces and it does.

I'm thinking more on the tcp wiresharks as I reproduce the issue a few more times.

So to recap we've known there were two issues at play (listed below). I believe now one of them isn't real and one is.

1) Pulp needs to add session reconnect/establishment for aiohttp's session. Their issue tracker recommends applications like Pulp carry this.

2) A strange TCP connection closure behavior where a single TCP stream being closed on the server normally would cause the client to initiate tcp closures on all other tcp connections to that server, e.g. comment 19.

I've realized that issue (1) is real and issue (2) is not!

To explain why we observed the behavior of issue (2), it's actually normal. It is expected that the client would immediately close all tcp connections because aiohttp encountered a fatal exception and Pulp left that unhandled. What could aiohttp do but shutdown? So it shuts down immediately, which closes all aiohttp sockets, causing the client -> server TCP RST segments that were observed in Comment 19.