Project

Profile

Help

Issue #8566

Content Migration to Pulp 3 with Katello fails (similar to #8377)

Added by cmaso 27 days ago. Updated 6 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

On my Katello system (Katello: 3.18.2, Foreman: 2.3.3) the content migration to Pulp3 fails with this message:

ForemanTasks::TaskError: Task 62598cf8-3107-4255-b321-1362c889b3a3: Katello::Errors::Pulp3Error: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=64f44866-0207-4005-9c06-0f45e52cbdd1>"

Installed Pulp modules are:

pulp-2to3-migration (0.11.0)
pulp-certguard (1.0.3)
pulp-container (2.1.0)
pulp-deb (2.7.0)
pulp-file (1.3.0)
pulp-rpm (3.10.0)
pulpcore (3.7.5)

All my repositories are configured to sync immediately.

Database contains the following for the mentioned content_id:

pulpcore=# select pulp_id, path, platforms from rpm_image where distribution_tree_id = '64f44866-0207-4005-9c06-0f45e52cbdd1';
               pulp_id                |           path            |  platforms
--------------------------------------+---------------------------+-------------
 9f630766-ffc7-4147-a8b5-133d1075b102 | images/boot.iso           | x86_64
 c84efc63-5963-49b9-889c-01e4d8518e25 | images/efiboot.img        | x86_64
 5d7e5127-ad71-4d2e-84d0-eeb0df61fbad | images/pxeboot/initrd.img | x86_64, xen
 630b295b-86f4-4fe4-a965-58f59ed19768 | images/pxeboot/vmlinuz    | x86_64, xen
(4 rows)
pulpcore=# select * from core_contentartifact where content_id = '64f44866-0207-4005-9c06-0f45e52cbdd1';
               pulp_id                |         pulp_created          |       pulp_last_updated       | relative_path |             artifact_id              |
        content_id
--------------------------------------+-------------------------------+-------------------------------+---------------+--------------------------------------+------
--------------------------------
 3023c31b-78be-4728-a770-6140d1550811 | 2021-04-15 10:10:56.816636+02 | 2021-04-15 10:10:56.816651+02 | .treeinfo     | e76343f3-3bab-4b7b-912f-954ff3a24714 | 64f44
866-0207-4005-9c06-0f45e52cbdd1
(1 row)

Thanks in advance!

History

#1 Updated by daviddavis 22 days ago

  • Project changed from Pulp to Migration Plugin

#2 Updated by ttereshc 22 days ago

Hi, could you provide full traceback from your logs? They should be available in the /var/log/messages and/or via journalctl.

I see the reference to #8377 but having your traceback might still be helpful. Thanks.

#3 Updated by cmaso 21 days ago

Sure, here it is:

Apr 21 07:36:18 stg-katello pulpcore-worker-3: /usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1427: RuntimeWarning: DateTimeField Pulp2RepoContent.pulp2_created received a naive datetime (2021-04-19 23:05:22) while time zone support is active.
Apr 21 07:36:18 stg-katello pulpcore-worker-3: RuntimeWarning)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 21 07:36:18 stg-katello pulpcore-worker-3: rv = job.perform()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 21 07:36:18 stg-katello pulpcore-worker-3: self._result = self._execute()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 21 07:36:18 stg-katello pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Apr 21 07:36:18 stg-katello pulpcore-worker-3: migrate_content(plan, skip_corrupted=skip_corrupted)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Apr 21 07:36:18 stg-katello pulpcore-worker-3: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Apr 21 07:36:18 stg-katello pulpcore-worker-3: loop.run_until_complete(dm.create())
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 21 07:36:18 stg-katello pulpcore-worker-3: return future.result()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Apr 21 07:36:18 stg-katello pulpcore-worker-3: await pipeline
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 21 07:36:18 stg-katello pulpcore-worker-3: await asyncio.gather(*futures)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 21 07:36:18 stg-katello pulpcore-worker-3: await self.run()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Apr 21 07:36:18 stg-katello pulpcore-worker-3: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 301, in _needed_remote_artifacts
Apr 21 07:36:18 stg-katello pulpcore-worker-3: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: ValueError: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=d0a0b30f-8cfc-4d76-96da-c2317f720819>"
Apr 21 07:36:18 stg-katello pulpcore-worker-3: Traceback (most recent call last):
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 21 07:36:18 stg-katello pulpcore-worker-3: rv = job.perform()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 21 07:36:18 stg-katello pulpcore-worker-3: self._result = self._execute()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 21 07:36:18 stg-katello pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Apr 21 07:36:18 stg-katello pulpcore-worker-3: migrate_content(plan, skip_corrupted=skip_corrupted)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Apr 21 07:36:18 stg-katello pulpcore-worker-3: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Apr 21 07:36:18 stg-katello pulpcore-worker-3: loop.run_until_complete(dm.create())
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 21 07:36:18 stg-katello pulpcore-worker-3: return future.result()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Apr 21 07:36:18 stg-katello pulpcore-worker-3: await pipeline
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 21 07:36:18 stg-katello pulpcore-worker-3: await asyncio.gather(*futures)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 21 07:36:18 stg-katello pulpcore-worker-3: await self.run()
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Apr 21 07:36:18 stg-katello pulpcore-worker-3: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Apr 21 07:36:18 stg-katello pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 301, in _needed_remote_artifacts
Apr 21 07:36:18 stg-katello pulpcore-worker-3: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Apr 21 07:36:18 stg-katello pulpcore-worker-3: ValueError: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=d0a0b30f-8cfc-4d76-96da-c2317f720819>"
Apr 21 07:36:18 stg-katello pulpcore-worker-3: pulp: rq.worker:INFO: Cleaning registries for queue: 44763@stg-katello.gtstg.lan
Apr 21 07:36:18 stg-katello pulpcore-worker-3: pulp: rq.worker:INFO: 44763@stg-katello.gtstg.lan: e26d231a-d1bc-49d6-aaea-de9b7fb59000
Apr 21 07:36:18 stg-katello pulpcore-worker-3: pulp: rq.worker:INFO: 44763@stg-katello.gtstg.lan: Job OK (e26d231a-d1bc-49d6-aaea-de9b7fb59000)

Don't know why the content id is different now, but when I (today) run it multiple times, the error now is always at this id.

#4 Updated by cmaso 21 days ago

I did some further testing today. With the above information I was able to find out which repository this content_id relates to: it was the High Availability repo. I've removed it from Katello and tried the migration again. Now it's again failing and the related repo is the PowerTools repo.

So for me it looks like I now need to remove all repos additionally to the Base repo, then I should be able to run the migration?

#5 Updated by jsherril@redhat.com 21 days ago

  • Tags Katello added

#6 Updated by ttereshc 21 days ago

Hi cmaso,

I'm trying to reproduce your issue, could you share the repos you are talking about? Preferably provide with the urls you synced them from. Are you referring to CentOS 8 repos? E.g. http://ftp.cvut.cz/centos/8/

Thanks!

#7 Updated by cmaso 20 days ago

Hi ttereshc,

yes, it's about the CentOS 8 repos. In my Katello I have synchronized the following:

For a test I removed all but BaseOS and then the preparation was successful (but that's currently only on my test system, the productive still has all of them and needs all of them)

#8 Updated by ttereshc 14 days ago

I ran into an issue when all the repos are on_demand, all additional repos apart from BaseOS are not migrated.
This is very Centos8 specific, because of how they construct those distribution trees.

If I make the repos immediate, then migration works fine for me (I'm on the same versions as mentioned in the description).

@cmaso, on your pulp2 system, do you have other distribution/kickstart repositories? or only centos8 ones? Do you have any content views in katello which are based on your centos8 repos? I'm trying to guess what can potentially be different, since I can't reproduce it.

#9 Updated by jsherril@redhat.com 14 days ago

Would you be able to upload your pulp2 database? It is a large file so you could use the red hat upload sftp server here: https://access.redhat.com/solutions/2112#secureftp

creating a tarball of /var/lib/mongodb/ and then uploading that would be helpful.

#10 Updated by gvde 12 days ago

I think I am hitting the same issue. Just discussing this at the foreman community

https://community.theforeman.org/t/contentmigration-katello-pulp3error/23331

#11 Updated by caseybea 12 days ago

I along with gvde am hitting this exact same issue.

What is quite likely bringing this to the forefront is Katello just released super brand-new version 4. And a requirement for eventually upgrading to that version requires that you migrate your pulp2 content to pulp3. I have been putting that off for a while, just attempted this week. and... hit this issue. Verified caused by Centos8 repos in m environment.

#12 Updated by cmaso 9 days ago

I ended up removing all my additional CentOS repos (kept only BaseOS) and then I was able to complete the migration.

After that I've updated Katello to 4.0 and added the other repos again and everything is now working (on Katello 4 and pulp3).

#13 Updated by caseybea 6 days ago

Sadly in my situation wiping out my centos8-like repos is not an option. I am deep in the middle of testing all of the popular Centos8 variants. Hoping for an update in the next week or so? I cannot migrate my katello to pulp3 currently. I do suspect more and more people will start hitting this.

Please register to edit this issue

Also available in: Atom PDF