Process Recycling feature causes HTTP event notifier to be unreliable
For some non-negligible probability, the HTTP event notifier will fail to send its notification when process recycling is enabled.
1. Enable Process Recycling
2. Configure the event notifier
3. Ensure (1) and (2) are both working as expected
4. Dispatch lots of tasks
5. Observe that not all events are notified
#2 Updated by bmbouter over 4 years ago
The root cause of this is the use of Daemon threads in the http event notifier. Here is the problematic workflow:
1. a pulp task starts running
2. an http notifier is requested with a call to event.http.handle_event
3. It spawns a thread "here:https://github.com/pulp/pulp/blob/a6f8f83514fb7255d540a61028a842860a8fb422/server/pulp/server/event/http.py#L33
4. The celery task completes process recycling occurs since it's enabled which kills the daemon thread since daemon threads won't hold a process open like non-daemon thread will.
5. The _send_post task may or may not complete because steps 2-4 occur faster than the other thread can reliably execute.
6. The message is never sent
#4 Updated by bmbouter over 4 years ago
To fix this I recommend we stop using threading at all here. I believe using threading here made sense when Pulp ran entirely in httpd in Pulp 2.3-. With the tasking system it makes no sense to try to send the notification asynchronously. We should combine all of the send logic into the handle_event() call and remove threading entirely from the http event notifier module.
#7 Updated by mhrivnak over 4 years ago
The reason threading was kept is so that an unresponsive web server could not stall Pulp tasks. Consider a web server that takes a long time to respond, or is in poor health and takes indefinitely. We need some way to make sure that does not cause Pulp tasks to grind to a halt.
Another way to do this, at least with the http notifier, is to make sure there is a reasonably short timeout on the request. I think that could be a reasonable alternative to the threading approach.