https://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-11-23T11:20:24ZPulpMigration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=650632020-11-23T11:20:24Zttereshcttereshc@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/65063/diff?detail_id=65175">diff</a>)</li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=651052020-11-24T17:14:04Zdalleydalley@redhat.com
<ul></ul><p>At the scales we are expecting, it is plausible that we could brute force this problem by:</p>
<ul>
<li>Extract all of the pulp 2 content IDs from mongo</li>
<li>Extract all of the pulp 2 content IDs from the premigrated content table in postgresql</li>
<li>Dump them into sets</li>
<li>Take a difference of the sets</li>
<li>Delete the pre-migrated content which was removed from Pulp 2</li>
</ul>
<p>MongoDB object IDs are 24 character long hexidecimal strings. 24 bytes x 2 lists x 4,000,000 content (which would be a <em>very</em> large installation) would yield about 200 megabytes of memory consumption. Since Python uses string interning and the contents of the lists are expected to be mostly duplicate, the actual number should be ~max 100 megabytes.</p>
<p>If we have to do this work anyway, we might also be able to use it to get rid of some corner cases.</p>
<p><a href="https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L140-L152" class="external">https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L140-L152</a>
<a href="https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L120-L124" class="external">https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L120-L124</a></p> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=665622021-01-19T19:16:30Zttereshcttereshc@redhat.com
<ul><li><strong>Sprint/Milestone</strong> set to <i>0.9.0</i></li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=675432021-02-12T14:06:18Zjsherril@redhat.comjsherril@redhat.com
<ul><li><strong>Tags</strong> <i>Katello</i> added</li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=680992021-03-02T17:08:08Zttereshcttereshc@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>ttereshc</i></li><li><strong>Sprint</strong> set to <i>Sprint 91</i></li><li><strong>Quarter</strong> set to <i>Q1-2021</i></li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=682062021-03-04T15:45:40Zttereshcttereshc@redhat.com
<ul></ul><p>This won't affect Katello much because now we have skip_corrupted=True and they always use it.
But it will be an extra work on every re-run.</p> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=683252021-03-05T20:09:35Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 91</i> to <i>Sprint 92</i></li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=683622021-03-05T21:30:11Zttereshcttereshc@redhat.com
<ul><li><strong>Sprint/Milestone</strong> deleted (<del><i>0.9.0</i></del>)</li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=688972021-03-19T20:25:39Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 92</i> to <i>Sprint 93</i></li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=689432021-03-22T17:58:04Zttereshcttereshc@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p><a href="https://github.com/pulp/pulp-2to3-migration/pull/341" class="external">https://github.com/pulp/pulp-2to3-migration/pull/341</a></p> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=689702021-03-23T17:47:40Zttereshcttereshc@redhat.com
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="Remove Pulp2Content records if content is no longer in Pulp2 If some content is only pre-migrate..." href="https://pulp.plan.io/projects/pulp/repository/pulp-2to3-migration/revisions/be75d4caabc5fe04a5acf331b2ece627cae463fa">pulp:pulp-2to3-migration|be75d4caabc5fe04a5acf331b2ece627cae463fa</a>.</p> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=689722021-03-23T17:57:44Zttereshcttereshc@redhat.com
<ul><li><strong>Sprint/Milestone</strong> set to <i>0.10.0</i></li></ul> Migration Plugin - Issue #7887: If orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raisedhttps://pulp.plan.io/issues/7887?journal_id=689782021-03-23T18:40:37Zpulpbot
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul>