Pulp

The root cause of this is the use of Daemon threads in the http event notifier. Here is the problematic workflow:

1. a pulp task starts running
2. an http notifier is requested with a call to event.http.handle_event
3. It spawns a thread "here:https://github.com/pulp/pulp/blob/a6f8f83514fb7255d540a61028a842860a8fb422/server/pulp/server/event/http.py#L33
4. The celery task completes process recycling occurs since it's enabled which kills the daemon thread since daemon threads won't hold a process open like non-daemon thread will.
5. The _send_post task may or may not complete because steps 2-4 occur faster than the other thread can reliably execute.
6. The message is never sent

To fix this I recommend we stop using threading at all here. I believe using threading here made sense when Pulp ran entirely in httpd in Pulp 2.3-. With the tasking system it makes no sense to try to send the notification asynchronously. We should combine all of the send logic into the handle_event() call and remove threading entirely from the http event notifier module.

that sounds good to me as well. It's already done in the task anyway so there's no harm in not threading.

The reason threading was kept is so that an unresponsive web server could not stall Pulp tasks. Consider a web server that takes a long time to respond, or is in poor health and takes indefinitely. We need some way to make sure that does not cause Pulp tasks to grind to a halt.

Another way to do this, at least with the http notifier, is to make sure there is a reasonably short timeout on the request. I think that could be a reasonable alternative to the threading approach.