https://pulp.plan.io/https://pulp.plan.io/favicon.ico2017-07-13T19:21:11ZPulpPulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208372017-07-13T19:21:11Zbmbouterbmbouter@redhat.com
<ul><li><strong>Subject</strong> changed from <i>HTTP event notifier leaks one thread with every message sent</i> to <i>Process Recycling feature causes HTTP event notifier to be unreliable</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/20837/diff?detail_id=21405">diff</a>)</li><li><strong>Priority</strong> changed from <i>Low</i> to <i>High</i></li><li><strong>Severity</strong> changed from <i>2. Medium</i> to <i>3. High</i></li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208402017-07-13T19:29:11Zbmbouterbmbouter@redhat.com
<ul></ul><p>The root cause of this is the use of Daemon threads in the http event notifier. Here is the problematic workflow:</p>
<p>1. a pulp task starts running<br>
2. an http notifier is requested with a call to <a href="https://github.com/pulp/pulp/blob/a6f8f83514fb7255d540a61028a842860a8fb422/server/pulp/server/event/http.py#L26" class="external">event.http.handle_event</a><br>
3. It spawns a thread "here:<a href="https://github.com/pulp/pulp/blob/a6f8f83514fb7255d540a61028a842860a8fb422/server/pulp/server/event/http.py#L33" class="external">https://github.com/pulp/pulp/blob/a6f8f83514fb7255d540a61028a842860a8fb422/server/pulp/server/event/http.py#L33</a><br>
4. The celery task completes process recycling occurs since it's enabled which kills the daemon thread since daemon threads won't hold a process open like non-daemon thread will.<br>
5. The _send_post task may or may not complete because steps 2-4 occur faster than the other thread can reliably execute.<br>
6. The message is never sent</p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208462017-07-13T19:56:33Zbmbouterbmbouter@redhat.com
<ul></ul><p>To fix this I recommend we stop using threading at all here. I believe using threading here made sense when Pulp ran entirely in httpd in Pulp 2.3-. With the tasking system it makes no sense to try to send the notification asynchronously. We should combine all of the send logic into the handle_event() call and remove threading entirely from the http event notifier module.</p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208472017-07-13T19:58:18Zcduryeecduryee@redhat.com
<ul></ul><p>that sounds good to me as well. It's already done in the task anyway so there's no harm in not threading.</p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208752017-07-14T14:38:39Zttereshcttereshc@redhat.com
<ul><li><strong>Sprint/Milestone</strong> set to <i>41</i></li><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208762017-07-14T14:39:04Zmhrivnakmhrivnak@redhat.com
<ul></ul><p>The reason threading was kept is so that an unresponsive web server could not stall Pulp tasks. Consider a web server that takes a long time to respond, or is in poor health and takes indefinitely. We need some way to make sure that does not cause Pulp tasks to grind to a halt.</p>
<p>Another way to do this, at least with the http notifier, is to make sure there is a reasonably short timeout on the request. I think that could be a reasonable alternative to the threading approach.</p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208882017-07-14T17:11:35Zpcreechpcreech@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>POST</i></li><li><strong>Assignee</strong> set to <i>pcreech</i></li></ul><p><a href="https://github.com/pulp/pulp/pull/3089" class="external">https://github.com/pulp/pulp/pull/3089</a></p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=208912017-07-14T18:06:15Zpcreechpcreech@redhat.com
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="Stop creating a thread for http notifier Stop creating a thread for http notifier closes #2903" href="https://pulp.plan.io/projects/pulp/repository/pulp/revisions/bd1c9206a9a66d30e216d820b46db47bb15b0faf">pulp|bd1c9206a9a66d30e216d820b46db47bb15b0faf</a>.</p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=211422017-07-20T18:40:33Zpcreechpcreech@redhat.com
<ul><li><strong>Platform Release</strong> set to <i>2.14.0</i></li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=212592017-07-27T15:15:41Zpcreechpcreech@redhat.com
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>5</i></li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=216262017-08-15T14:17:58Zpcreechpcreech@redhat.com
<ul><li><strong>Status</strong> changed from <i>5</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=231972017-12-06T15:42:58Zkersom
<ul></ul><p>This issue was manually verified.</p>
<p>See: <a href="https://github.com/PulpQE/pulp-smash/issues/714" class="external">Smash 714</a></p> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=260712018-03-08T23:18:41Zbmbouterbmbouter@redhat.com
<ul><li><strong>Sprint</strong> set to <i>Sprint 22</i></li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=260962018-03-08T23:20:03Zbmbouterbmbouter@redhat.com
<ul><li><strong>Sprint/Milestone</strong> deleted (<del><i>41</i></del>)</li></ul> Pulp - Issue #2903: Process Recycling feature causes HTTP event notifier to be unreliablehttps://pulp.plan.io/issues/2903?journal_id=381952019-04-15T20:16:49Zbmbouterbmbouter@redhat.com
<ul><li><strong>Tags</strong> <i>Pulp 2</i> added</li></ul>