Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #7193

closed

Packages from Fedora repositories not migrating properly

Added by dalley over 3 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
dalley
Sprint/Milestone:
0.4.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Pulp 2

Id:                  centos7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:        1
  Package Category:    11
  Package Environment: 10
  Package Group:       88
  Package Langpacks:   1
  Rpm:                 10070

Id:                  centos6
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:     1
  Package Category: 14
  Package Group:    214
  Rpm:              6713

Id:                  centos8-baseos
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:        1
  Package Category:    4
  Package Environment: 3
  Package Group:       35
  Rpm:                 1675

Id:                  centos8-appstream
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:        1
  Modulemd:            66
  Modulemd Defaults:   42
  Package Category:    5
  Package Environment: 4
  Package Group:       58
  Rpm:                 5359

Id:                  fedora30-updates
Display Name:        None
Description:         None
Content Unit Counts: 
  Drpm:                30
  Erratum:             3970
  Package Category:    15
  Package Environment: 17
  Package Group:       149
  Package Langpacks:   1
  Rpm:                 16235

Id:                  fedora31-updates
Display Name:        None
Description:         None
Content Unit Counts: 
  Drpm:                68
  Erratum:             4097
  Package Category:    15
  Package Environment: 18
  Package Group:       152
  Package Langpacks:   1
  Rpm:                 21337

Pulp 3

In [1]: Package.objects.count()                                                                                                                                                                                                               
Out[1]: 27109

In [2]: Pulp2Content.objects.count()                                                                                                                                                                                                          
Out[2]: 70383

In [3]: for repo in Repository.objects.all(): 
   ...:     lv = repo.latest_version() 
   ...:     print(f"{repo.name}: {lv.content.filter(pulp_type=Package.get_pulp_type()).count()}") 
   ...:                                                                                                                                                                                                                                       
fedora31-updates: 69
fedora30-updates: 3223
centos6: 6713
centos7: 10070
centos8-appstream: 5359
centos8-baseos: 1675

The Packages from the centos repositories are migrated properly, but not from the fedora repositories. On multiple migration re-runs the result is the same each time so it is deterministically wrong.

I think the fedora30 and 31 "release" repos are also migrated incorrectly though I don't have them synced at the moment to test. I tested them on a previous box and didn't record the measurements.

Related issues

Related to Migration Plugin - Issue #6469: exceptions encountered during migration run are not captured in a task CLOSED - CURRENTRELEASEdalleyActions
Actions
Copy link
 #1

Updated by dalley over 3 years ago

  • Subject changed from Fedora repositories not migrating properly to Packages from Fedora repositories not migrating properly
Actions
Copy link
 #2

Updated by dalley over 3 years ago

I'm seeing a bunch of this in the worker logs:

Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dbe35aca-9125-4f26-b55a-c28c0a0e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dc00a508-c3d6-445f-95cf-0b10f169
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dc67c34e-921e-497e-ab6b-38988f25
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dcb2fd31-f8da-45f8-b5b3-b08e2dc3
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dcdce325-dd08-4755-9e97-6db6d76e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd02aa4e-cf38-465b-a19a-1d78f1e9
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd45a1b1-c59a-47a8-b043-b3b7da13
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dd5621d8-1bef-4022-8f0f-8f495798
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: ddef1236-5528-4700-a533-d76ad073
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: de1c7991-42a9-41ec-8cfa-72debdd9
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: de36ea68-749d-4e4a-84e6-8f43b4a6
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dea0419e-f303-481f-8a32-e01ac6b1
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: deaabc7d-ad5f-43f8-aa13-3258a550
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: def1c8ef-6ea8-43eb-8366-8147399e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e347a83b-540b-43a9-9d22-4903dd90
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df1c527c-5dec-43fa-8f94-c82eeb8e
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df31a234-f5bf-4385-ad2e-01c05580
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: df88721f-b12f-45aa-8b7c-51325cb7
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfa6e938-68a8-4fc7-95a9-9a07d665
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfc193c7-407e-4449-9dd7-5dba811d
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: dfd00489-701a-41f9-b21f-3775130b
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e06b8955-d01c-40be-a866-3b1511a4
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e3a9a02c-7ac2-4a4b-a304-4977f0c7
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e467ae9a-ac36-4941-88ac-27ae4eda
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e4769ead-9945-4a09-92dc-2db7b8d5
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e4b86caf-3c09-4b60-85bf-05895677
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e51448f9-a718-43f9-a2a9-2c3f99b2
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e0963af4-e34c-4db1-b438-e14c3f9d
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e184bea9-286b-4e24-959a-93e536a6
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e1be787b-6ed1-4234-9175-bc227711
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e1ed17cb-7ecd-4be3-b6cb-daaccd82
Jul 22 13:09:46 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[3279]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without a remote pulp2 unit_id: e2134e3b-71fb-4ae2-b7be-4a660371
Actions
Copy link
 #3

Updated by dalley over 3 years ago

Also possibly relevant, but could be an entirely separate issue:

edit: I think it's a separate issue that we need to file with createrepo_c.

 cr_xml_parser_generic_from_string: parsing error '<otherdata><package arch="noarch" name="flat-remix-icon-theme" pkgid="0140bd17f108c6d272ec953885ad5e03a9e
                                                                                     <version epoch="0" rel="1.fc30" ver="0.0.20191223" />


.... snip lots and lots and lots of XML ...  thousands and thousands of <file> tags for various icons


                                                                                     <file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-severe-alert-symbolic.svg</file>
                                                                                     <file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-showers-scattered-symbolic.svg</file>
                                                                                     <file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-showers-symbolic.svg</file>
                                                                                     <file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-snow-symbolic.svg</file>
                                                                                     <file>/usr/share/icons/Flat-Remix-Yellow/status/symbolic/weather-storm-symbolic.svg</file>
                                                                                     <file type="dir">/usr/share/licenses/flat-remix-icon-theme</file>
                                                                                     <file>/usr/share/licenses/flat-remix-icon-theme/LICENSE</file>
                                                                                   </package>
                                                                                   </otherdata>': internal error: Huge input lookup
Actions
Copy link
 #4

Updated by ttereshc over 3 years ago

I think there are 2 possible cases for the warning about on_demand content not being migrated:

  1. None of the importers from those repositories are specified in the Migration Plan.
    • No on_demand content will be migrated.
  2. Content has multiple importers and at least one of them is not specified in the Migration plan.
    • RemoteArtifacts won't be created for the urls from the missed importers.
    • Content should be migrated if at least one relevant importer is migrated.
    • There is a room for improvement here, we should not send warning in such case. So introducing a flag which will tell if at least one importer/remote was found can help here.

Relevant piece of code https://github.com/pulp/pulp-2to3-migration/blob/a29b2f08890c41be0de90709cd772e778d96e019/pulp_2to3_migration/app/plugin/content.py#L343-L349

Actions
Copy link
 #5

Updated by dalley over 3 years ago

So, I found some nondeterminism :(

If you migrate the very same Pulp 2 repo over and over, you always get the same result for the # of Packages properly migrated. However, if you reset the Pulp 2 database and then re-sync the repository with exactly the same options, and then migrate again, you get a different result....

Actions
Copy link
 #6

Updated by dalley over 3 years ago

The createrepo_c issue is causing this issue. Once I removed the content unit from the system (from repo, then orphan cleanup), migrations work again. We can close this it once that issue is resolved.

Marking https://pulp.plan.io/issues/6469 as related because the error not being bubbled up is why it snuck under the radar.

Actions
Copy link
 #7

Updated by dalley over 3 years ago

  • Related to Issue #6469: exceptions encountered during migration run are not captured in a task added
Actions
Copy link
 #8

Updated by dalley over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

Added by ttereshc over 3 years ago

Revision 0710f1e2 | View on GitHub

Don't log any warnings if at least one LCE has been migrated.

re #7193 https://pulp.plan.io/issues/7193

Added by ttereshc over 3 years ago

Revision 0710f1e2 | View on GitHub

Don't log any warnings if at least one LCE has been migrated.

re #7193 https://pulp.plan.io/issues/7193

Added by ttereshc over 3 years ago

Revision 0710f1e2 | View on GitHub

Don't log any warnings if at least one LCE has been migrated.

re #7193 https://pulp.plan.io/issues/7193

Actions
Copy link
 #9

Updated by dalley over 3 years ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE
Actions
Copy link
 #10

Updated by adam.winberg@smhi.se over 3 years ago

dalley wrote:

The createrepo_c issue is causing this issue. Once I removed the content unit from the system (from repo, then orphan cleanup), migrations work again. We can close this it once that issue is resolved.

Marking https://pulp.plan.io/issues/6469 as related because the error not being bubbled up is why it snuck under the radar.

I'm also seeing this createrepo_c issue ('Huge input lookup'), it doesn't look like that was resolved in this issue?

Actions
Copy link
 #11

Updated by dalley over 3 years ago

Hi Adam, which version of createrepo_c are you using and how is it installed? If you're using a Python package-based installation and you haven't already, make sure to update the createrepo_c package with "pip install --upgrade createrepo-c".

Actions
Copy link
 #12

Updated by adam.winberg@smhi.se over 3 years ago

dalley wrote:

Hi Adam, which version of createrepo_c are you using and how is it installed? If you're using a Python package-based installation and you haven't already, make sure to update the createrepo_c package with "pip install --upgrade createrepo-c".

I'm using a rpm-based installation with python3-createrepo_c-0.15.10-1.el8.x86_64 from the katello repos.

Actions
Copy link
 #13

Updated by dalley over 3 years ago

The RPMs must be missing the patch for that fix, then. I'll ask that they be updated.

Actions
Copy link
 #14

Updated by ttereshc over 3 years ago

  • Status changed from CLOSED - CURRENTRELEASE to MODIFIED
Actions
Copy link
 #15

Updated by ttereshc over 3 years ago

  • Sprint/Milestone set to 0.4.0
Actions
Copy link
 #16

Updated by pulpbot over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Copy link
 #17

Updated by adam.winberg@smhi.se over 3 years ago

There is still no updated version of python3-createrepo_c in the katello yum repositories.

Actions
Send by e-mail Copy link

Also available in: Atom PDF