https://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-07-22T16:18:12ZPulpMigration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598022020-07-22T16:18:12Zdalleydalley@redhat.com
<ul><li><strong>Subject</strong> changed from <i>Fedora repositories not migrating properly</i> to <i>Packages from Fedora repositories not migrating properly</i></li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598042020-07-22T16:22:35Zdalleydalley@redhat.com
<ul></ul><p>I'm seeing a bunch of this in the worker logs:</p>
<pre><code>Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dbe35aca-9125-4f26-b55a-c28c0a0e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dc00a508-c3d6-445f-95cf-0b10f169
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dc67c34e-921e-497e-ab6b-38988f25
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dcb2fd31-f8da-45f8-b5b3-b08e2dc3
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dcdce325-dd08-4755-9e97-6db6d76e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd02aa4e-cf38-465b-a19a-1d78f1e9
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd45a1b1-c59a-47a8-b043-b3b7da13
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd5621d8-1bef-4022-8f0f-8f495798
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: ddef1236-5528-4700-a533-d76ad073
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: de1c7991-42a9-41ec-8cfa-72debdd9
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: de36ea68-749d-4e4a-84e6-8f43b4a6
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dea0419e-f303-481f-8a32-e01ac6b1
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: deaabc7d-ad5f-43f8-aa13-3258a550
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: def1c8ef-6ea8-43eb-8366-8147399e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e347a83b-540b-43a9-9d22-4903dd90
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df1c527c-5dec-43fa-8f94-c82eeb8e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df31a234-f5bf-4385-ad2e-01c05580
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df88721f-b12f-45aa-8b7c-51325cb7
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfa6e938-68a8-4fc7-95a9-9a07d665
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfc193c7-407e-4449-9dd7-5dba811d
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfd00489-701a-41f9-b21f-3775130b
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e06b8955-d01c-40be-a866-3b1511a4
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e3a9a02c-7ac2-4a4b-a304-4977f0c7
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e467ae9a-ac36-4941-88ac-27ae4eda
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e4769ead-9945-4a09-92dc-2db7b8d5
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e4b86caf-3c09-4b60-85bf-05895677
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e51448f9-a718-43f9-a2a9-2c3f99b2
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e0963af4-e34c-4db1-b438-e14c3f9d
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e184bea9-286b-4e24-959a-93e536a6
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e1be787b-6ed1-4234-9175-bc227711
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e1ed17cb-7ecd-4be3-b6cb-daaccd82
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e2134e3b-71fb-4ae2-b7be-4a660371
</code></pre> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598052020-07-22T16:36:21Zdalleydalley@redhat.com
<ul></ul><p>Also <em>possibly</em> relevant, but could be an entirely separate issue:</p>
<p>edit: I think it's a separate issue that we need to file with createrepo_c.</p>
<pre><code> cr_xml_parser_generic_from_string: parsing error '<otherdata><package arch="noarch" name="flat-remix-icon-theme" pkgid="0140bd17f108c6d272ec953885ad5e03a9e
<version epoch="0" rel="1.fc30" ver="0.0.20191223" />
.... snip lots and lots and lots of XML ... thousands and thousands of <file> tags for various icons
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-severe-alert-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-showers-scattered-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-showers-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-snow-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-storm-symbolic.svg</file>
<file type="dir">/usr/share/licenses/flat-remix-icon-theme</file>
<file>/usr/share/licenses/flat-remix-icon-theme/LICENSE</file>
</package>
</otherdata>': internal error: Huge input lookup
</code></pre> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598122020-07-22T17:45:27Zttereshcttereshc@redhat.com
<ul></ul><p>I think there are 2 possible cases for the warning about on_demand content not being migrated:</p>
<ol>
<li>None of the importers from those repositories are specified in the Migration Plan.
<ul>
<li>No on_demand content will be migrated.</li>
</ul>
</li>
<li>Content has multiple importers and at least one of them is not specified in the Migration plan.
<ul>
<li>RemoteArtifacts won't be created for the urls from the missed importers.</li>
<li>Content should be migrated if at least one relevant importer is migrated.</li>
<li>There is a room for improvement here, we should not send warning in such case. So introducing a flag which will tell if at least one importer/remote was found can help here.</li>
</ul>
</li>
</ol>
<p>Relevant piece of code <a href="https://github.com/pulp/pulp-2to3-migration/blob/a29b2f08890c41be0de90709cd772e778d96e019/pulp_2to3_migration/app/plugin/content.py#L343-L349" class="external">https://github.com/pulp/pulp-2to3-migration/blob/a29b2f08890c41be0de90709cd772e778d96e019/pulp_2to3_migration/app/plugin/content.py#L343-L349</a></p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598192020-07-23T04:50:36Zdalleydalley@redhat.com
<ul></ul><p>So, I found some nondeterminism :(</p>
<p>If you migrate the very same Pulp 2 repo over and over, you always get the same result for the # of Packages properly migrated. However, if you reset the Pulp 2 database and then re-sync the repository with exactly the same options, and then migrate again, you get a different result....</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598502020-07-23T17:26:00Zdalleydalley@redhat.com
<ul></ul><p>The createrepo_c issue is causing this issue. Once I removed the content unit from the system (from repo, then orphan cleanup), migrations work again. We can close this it once that issue is resolved.</p>
<p>Marking <a href="https://pulp.plan.io/issues/6469" class="external">https://pulp.plan.io/issues/6469</a> as related because the error not being bubbled up is why it snuck under the radar.</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=598522020-07-23T17:26:18Zdalleydalley@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-11 priority-7 priority-high2 closed" href="/issues/6469">Issue #6469</a>: exceptions encountered during migration run are not captured in a task </i> added</li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=599112020-07-24T15:36:39Zdalleydalley@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>dalley</i></li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=619602020-09-04T03:00:11Zdalleydalley@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=629402020-09-25T13:24:01Zadam.winberg@smhi.se
<ul></ul><p>dalley wrote:</p>
<blockquote>
<p>The createrepo_c issue is causing this issue. Once I removed the content unit from the system (from repo, then orphan cleanup), migrations work again. We can close this it once that issue is resolved.</p>
<p>Marking <a href="https://pulp.plan.io/issues/6469" class="external">https://pulp.plan.io/issues/6469</a> as related because the error not being bubbled up is why it snuck under the radar.</p>
</blockquote>
<p>I'm also seeing this createrepo_c issue ('Huge input lookup'), it doesn't look like that was resolved in this issue?</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=629652020-09-25T13:52:10Zdalleydalley@redhat.com
<ul></ul><p>Hi Adam, which version of createrepo_c are you using and how is it installed? If you're using a Python package-based installation and you haven't already, make sure to update the createrepo_c package with "pip install --upgrade createrepo-c".</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=630342020-09-28T07:03:16Zadam.winberg@smhi.se
<ul></ul><p>dalley wrote:</p>
<blockquote>
<p>Hi Adam, which version of createrepo_c are you using and how is it installed? If you're using a Python package-based installation and you haven't already, make sure to update the createrepo_c package with "pip install --upgrade createrepo-c".</p>
</blockquote>
<p>I'm using a rpm-based installation with python3-createrepo_c-0.15.10-1.el8.x86_64 from the katello repos.</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=630392020-09-28T14:51:43Zdalleydalley@redhat.com
<ul></ul><p>The RPMs must be missing the patch for that fix, then. I'll ask that they be updated.</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=633262020-10-07T07:56:45Zttereshcttereshc@redhat.com
<ul><li><strong>Status</strong> changed from <i>CLOSED - CURRENTRELEASE</i> to <i>MODIFIED</i></li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=633352020-10-07T07:59:09Zttereshcttereshc@redhat.com
<ul><li><strong>Sprint/Milestone</strong> set to <i>0.4.0</i></li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=633542020-10-07T11:47:20Zpulpbot
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=634052020-10-08T05:12:47Zadam.winberg@smhi.se
<ul></ul><p>There is still no updated version of python3-createrepo_c in the katello yum repositories.</p> Migration Plugin - Issue #7193: Packages from Fedora repositories not migrating properlyhttps://pulp.plan.io/issues/7193?journal_id=636092020-10-12T13:54:07Zdalleydalley@redhat.com
<ul></ul><p>@Adam, it looks like it was indeed updated, but only for the 3.7 repository. I'll ask if they can publish the new RPMs for the older versions as well.</p>
<p>In the meantime, you can install them directly from the 3.7 repository</p>
<ul>
<li><a href="https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/createrepo_c-0.16.1-1.el8.x86_64.rpm" class="external">https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/createrepo_c-0.16.1-1.el8.x86_64.rpm</a></li>
<li><a href="https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/createrepo_c-libs-0.16.1-1.el8.x86_64.rpm" class="external">https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/createrepo_c-libs-0.16.1-1.el8.x86_64.rpm</a></li>
<li><a href="https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/createrepo_c-devel-0.16.1-1.el8.x86_64.rpm" class="external">https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/createrepo_c-devel-0.16.1-1.el8.x86_64.rpm</a></li>
<li><a href="https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/python3-createrepo_c-0.16.1-1.el8.x86_64.rpm" class="external">https://yum.theforeman.org/pulpcore/3.7/el8/x86_64/python3-createrepo_c-0.16.1-1.el8.x86_64.rpm</a></li>
</ul>