https://pulp.plan.io/https://pulp.plan.io/favicon.ico2018-10-16T22:20:08ZPulpPulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314172018-10-16T22:20:08Zjortel@redhat.comjortel@redhat.com
<ul><li><strong>Tags</strong> <i>Pulp 3</i> added</li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314742018-10-19T14:51:50ZCodeHeeler
<ul><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li><li><strong>Sprint</strong> set to <i>Sprint 44</i></li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314862018-10-19T15:41:26Zbmbouterbmbouter@redhat.com
<ul></ul><p>This is totally different on the single-content branch which converts the last call to bulk_create() and all of them occur in the transaction. So I think this is maybe NOTABUG. I missed triage so I can only comment now.</p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314872018-10-19T15:42:58Zjortel@redhat.comjortel@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/31487/diff?detail_id=32326">diff</a>)</li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314902018-10-19T16:07:53Zdaviddavis
<ul><li><strong>Sprint</strong> deleted (<del><i>Sprint 44</i></del>)</li></ul><p>Removing from the sprint due to lack of agreement on around how to handle this.</p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314912018-10-19T16:11:32Zjortel@redhat.comjortel@redhat.com
<ul></ul><p>bmbouter wrote:</p>
<blockquote>
<p>This is totally different on the single-content branch which converts the last call to bulk_create() and all of them occur in the transaction. So I think this is maybe NOTABUG. I missed triage so I can only comment now.</p>
</blockquote>
<p>What do you mean by <em>"single-content branch"</em> ?</p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314932018-10-19T16:12:19Zamacdona@redhat.comaustin@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-11 priority-6 priority-default closed" href="/issues/4060">Issue #4060</a>: QueryExistingArtifacts stage does not prevent duplicates within a stream</i> added</li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314952018-10-19T16:19:15Zamacdona@redhat.comaustin@redhat.com
<ul></ul><p>There has been some confusion with this one and <a href="https://pulp.plan.io/issues/4060" class="external">https://pulp.plan.io/issues/4060</a>. 4060 is a problem within a single task (sync) of duplicates existing in a stream at the same time. This problem is related, but distinct. As noted in the description, this problem exists when 2 or more tasks are happening at once. If a unit passes through QueryExisting as a non-dupe, and while it is waiting in the Queue for the save stage, and another task creates that unit, it this one becomes a dupe and will fail when it is bulk_saved.</p>
<p>I think it is important to document these problems separately since they occur for different reasons. Some possible solutions might fix both at once. For instance, with docker sync we solve both problems by implementing our own save stage which does not use bulk writes. This allows us to catch and handle integrity errors as they happen. Adding this stage to pulpcore would solve both problems at the expense of database write performance. Since db performance could be a bottleneck, it is worth considering other options if we have any-- but those options would need to take into account both problems (dupes in stream, parallel tasks).</p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314962018-10-19T16:21:58Zjortel@redhat.comjortel@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/1">bmbouter</a>, What makes you think that this cannot happen?</p>
<pre><code>worker 1: query content ABC (QueryStage)
not-found
worker 2: query content ABC (QueryStage)
not-found
worker 1: create content ABC (CreateStage)
Inserted
worker 2: create content ABC (CreateStage)
IntegrityError
</code></pre> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=314982018-10-19T18:40:06Zdaviddavis
<ul></ul><p>@jortel, I agree it could happen. We saw the same problem when we worked on base_paths. We thought running both the query and the create in a single transaction would solve it but it did not:</p>
<p><a href="https://github.com/pulp/pulp/pull/3380#issuecomment-380477101" class="external">https://github.com/pulp/pulp/pull/3380#issuecomment-380477101</a></p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=315512018-10-29T16:49:39Zbmbouterbmbouter@redhat.com
<ul></ul><p>Recapping some of the discussion from last week, we do need to handle IntegrityErrors-with-requery support in the ContentUnitSaver stage. This ticket can track that. We also need to handle it in the ArtifactSaver stage too, which I believe is tracked as issue <a href="https://pulp.plan.io/issues/4060" class="external">https://pulp.plan.io/issues/4060</a></p>
<p>Does that sound right ^?</p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=315532018-10-29T17:58:15Zdaviddavis
<ul></ul><p>I thought @jortel was going to work on a general solution that could be applied to anywhere we call bulk_create and write a ticket for it.</p> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=320792018-11-21T18:14:10Zamacdona@redhat.comaustin@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-11 priority-6 priority-default closed" href="/issues/4178">Refactor #4178</a>: Update sync to use ContentSaver Stage</i> added</li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=321812018-11-28T15:11:46Zdkliban@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>MODIFIED</i></li><li><strong>Assignee</strong> set to <i>dkliban@redhat.com</i></li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=413452019-04-25T16:44:53Zdaviddavis
<ul><li><strong>Sprint/Milestone</strong> set to <i>3.0.0</i></li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=422832019-04-26T20:33:40Zbmbouterbmbouter@redhat.com
<ul><li><strong>Tags</strong> deleted (<del><i>Pulp 3</i></del>)</li></ul> Pulp - Issue #4085: ContentUnitSaver stage is vulnerable to race conditions.https://pulp.plan.io/issues/4085?journal_id=512852019-12-13T17:34:27Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul>