Issue #7193
closed
Packages from Fedora repositories not migrating properly
Status:
CLOSED - CURRENTRELEASE
Description
Pulp 2
Id: centos7
Display Name: None
Description: None
Content Unit Counts:
Distribution: 1
Package Category: 11
Package Environment: 10
Package Group: 88
Package Langpacks: 1
Rpm: 10070
Id: centos6
Display Name: None
Description: None
Content Unit Counts:
Distribution: 1
Package Category: 14
Package Group: 214
Rpm: 6713
Id: centos8-baseos
Display Name: None
Description: None
Content Unit Counts:
Distribution: 1
Package Category: 4
Package Environment: 3
Package Group: 35
Rpm: 1675
Id: centos8-appstream
Display Name: None
Description: None
Content Unit Counts:
Distribution: 1
Modulemd: 66
Modulemd Defaults: 42
Package Category: 5
Package Environment: 4
Package Group: 58
Rpm: 5359
Id: fedora30-updates
Display Name: None
Description: None
Content Unit Counts:
Drpm: 30
Erratum: 3970
Package Category: 15
Package Environment: 17
Package Group: 149
Package Langpacks: 1
Rpm: 16235
Id: fedora31-updates
Display Name: None
Description: None
Content Unit Counts:
Drpm: 68
Erratum: 4097
Package Category: 15
Package Environment: 18
Package Group: 152
Package Langpacks: 1
Rpm: 21337
Pulp 3
In [1]: Package.objects.count()
Out[1]: 27109
In [2]: Pulp2Content.objects.count()
Out[2]: 70383
In [3]: for repo in Repository.objects.all():
...: lv = repo.latest_version()
...: print(f"{repo.name}: {lv.content.filter(pulp_type=Package.get_pulp_type()).count()}")
...:
fedora31-updates: 69
fedora30-updates: 3223
centos6: 6713
centos7: 10070
centos8-appstream: 5359
centos8-baseos: 1675
The Packages from the centos repositories are migrated properly, but not from the fedora repositories. On multiple migration re-runs the result is the same each time so it is deterministically wrong.
I think the fedora30 and 31 "release" repos are also migrated incorrectly though I don't have them synced at the moment to test. I tested them on a previous box and didn't record the measurements.
- Subject changed from Fedora repositories not migrating properly to Packages from Fedora repositories not migrating properly
I'm seeing a bunch of this in the worker logs:
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dbe35aca-9125-4f26-b55a-c28c0a0e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dc00a508-c3d6-445f-95cf-0b10f169
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dc67c34e-921e-497e-ab6b-38988f25
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dcb2fd31-f8da-45f8-b5b3-b08e2dc3
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dcdce325-dd08-4755-9e97-6db6d76e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd02aa4e-cf38-465b-a19a-1d78f1e9
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd45a1b1-c59a-47a8-b043-b3b7da13
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd5621d8-1bef-4022-8f0f-8f495798
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: ddef1236-5528-4700-a533-d76ad073
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: de1c7991-42a9-41ec-8cfa-72debdd9
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: de36ea68-749d-4e4a-84e6-8f43b4a6
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dea0419e-f303-481f-8a32-e01ac6b1
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: deaabc7d-ad5f-43f8-aa13-3258a550
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: def1c8ef-6ea8-43eb-8366-8147399e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e347a83b-540b-43a9-9d22-4903dd90
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df1c527c-5dec-43fa-8f94-c82eeb8e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df31a234-f5bf-4385-ad2e-01c05580
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df88721f-b12f-45aa-8b7c-51325cb7
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfa6e938-68a8-4fc7-95a9-9a07d665
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfc193c7-407e-4449-9dd7-5dba811d
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfd00489-701a-41f9-b21f-3775130b
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e06b8955-d01c-40be-a866-3b1511a4
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e3a9a02c-7ac2-4a4b-a304-4977f0c7
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e467ae9a-ac36-4941-88ac-27ae4eda
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e4769ead-9945-4a09-92dc-2db7b8d5
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e4b86caf-3c09-4b60-85bf-05895677
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e51448f9-a718-43f9-a2a9-2c3f99b2
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e0963af4-e34c-4db1-b438-e14c3f9d
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e184bea9-286b-4e24-959a-93e536a6
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e1be787b-6ed1-4234-9175-bc227711
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e1ed17cb-7ecd-4be3-b6cb-daaccd82
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e2134e3b-71fb-4ae2-b7be-4a660371
Also possibly relevant, but could be an entirely separate issue:
edit: I think it's a separate issue that we need to file with createrepo_c.
cr_xml_parser_generic_from_string: parsing error '<otherdata><package arch="noarch" name="flat-remix-icon-theme" pkgid="0140bd17f108c6d272ec953885ad5e03a9e
<version epoch="0" rel="1.fc30" ver="0.0.20191223" />
.... snip lots and lots and lots of XML ... thousands and thousands of <file> tags for various icons
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-severe-alert-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-showers-scattered-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-showers-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-snow-symbolic.svg</file>
<file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-storm-symbolic.svg</file>
<file type="dir">/usr/share/licenses/flat-remix-icon-theme</file>
<file>/usr/share/licenses/flat-remix-icon-theme/LICENSE</file>
</package>
</otherdata>': internal error: Huge input lookup
I think there are 2 possible cases for the warning about on_demand content not being migrated:
- None of the importers from those repositories are specified in the Migration Plan.
- No on_demand content will be migrated.
- Content has multiple importers and at least one of them is not specified in the Migration plan.
- RemoteArtifacts won't be created for the urls from the missed importers.
- Content should be migrated if at least one relevant importer is migrated.
- There is a room for improvement here, we should not send warning in such case. So introducing a flag which will tell if at least one importer/remote was found can help here.
Relevant piece of code https://github.com/pulp/pulp-2to3-migration/blob/a29b2f08890c41be0de90709cd772e778d96e019/pulp_2to3_migration/app/plugin/content.py#L343-L349
So, I found some nondeterminism :(
If you migrate the very same Pulp 2 repo over and over, you always get the same result for the # of Packages properly migrated. However, if you reset the Pulp 2 database and then re-sync the repository with exactly the same options, and then migrate again, you get a different result....
The createrepo_c issue is causing this issue. Once I removed the content unit from the system (from repo, then orphan cleanup), migrations work again. We can close this it once that issue is resolved.
Marking https://pulp.plan.io/issues/6469 as related because the error not being bubbled up is why it snuck under the radar.
- Related to Issue #6469: exceptions encountered during migration run are not captured in a task added
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
- Status changed from ASSIGNED to CLOSED - CURRENTRELEASE
dalley wrote:
The createrepo_c issue is causing this issue. Once I removed the content unit from the system (from repo, then orphan cleanup), migrations work again. We can close this it once that issue is resolved.
Marking https://pulp.plan.io/issues/6469 as related because the error not being bubbled up is why it snuck under the radar.
I'm also seeing this createrepo_c issue ('Huge input lookup'), it doesn't look like that was resolved in this issue?
Hi Adam, which version of createrepo_c are you using and how is it installed? If you're using a Python package-based installation and you haven't already, make sure to update the createrepo_c package with "pip install --upgrade createrepo-c".
dalley wrote:
Hi Adam, which version of createrepo_c are you using and how is it installed? If you're using a Python package-based installation and you haven't already, make sure to update the createrepo_c package with "pip install --upgrade createrepo-c".
I'm using a rpm-based installation with python3-createrepo_c-0.15.10-1.el8.x86_64 from the katello repos.
The RPMs must be missing the patch for that fix, then. I'll ask that they be updated.
- Status changed from CLOSED - CURRENTRELEASE to MODIFIED
- Sprint/Milestone set to 0.4.0
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
There is still no updated version of python3-createrepo_c in the katello yum repositories.
@Adam, it looks like it was indeed updated, but only for the 3.7 repository. I'll ask if they can publish the new RPMs for the older versions as well.
In the meantime, you can install them directly from the 3.7 repository
Also available in: Atom
PDF
Don't log any warnings if at least one LCE has been migrated.
re #7193 https://pulp.plan.io/issues/7193