Pulp

Hi Florian, did you mean "http://mirror.centos.org/centos/8/BaseOS/x86_64/os/" for the repo URL?

Also here's the link to the Foreman community forum thread that preceded this bug: https://community.theforeman.org/t/katello-3-18-1-content-migration-to-pulp-3-fails-with-no-declared-artifact-message/22645/2

Oops, I indeed meant to write "http://mirror.centos.org/centos/8/BaseOS/x86_64/os/" for the repo URL. Sorry...

@fbachmann - what does the following show on the pulp3 side of the world?

select pulp_id, path, platforms from rpm_image where distribution_tree_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';

On my pulp3 instance after sync'ing CentOS8-Base, I see the following:

pulp=> select pulp_id, path, platforms from rpm_image where distribution_tree_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
               pulp_id                |           path            |  platforms  
--------------------------------------+---------------------------+-------------
 1360f187-5e59-4337-8368-2859d3fc83d7 | images/pxeboot/vmlinuz    | x86_64, xen
 99b1e311-4fa3-4661-a7ce-6586366d8b16 | images/pxeboot/initrd.img | x86_64, xen
 5e29d099-ff40-4203-acde-b76442af5dea | images/efiboot.img        | x86_64
 83f2e0f1-6f05-4618-be98-232078df5180 | images/boot.iso           | x86_64
(4 rows)

pulp=> 

I'm going to add "sync to pulp2, and migrate" into the experiment next.

Also - on the Pulp2 side of things - is your CentOS8-Base sync'd immediate or on_demand?

ggainey wrote:

@fbachmann - what does the following show on the pulp3 side of the world?

select pulp_id, path, platforms from rpm_image where distribution_tree_id = '0de942aa-2116-454c-9235-bbdcad17b488';

On my pulp3 instance after sync'ing CentOS8-Base, I see the following:

pulp=> select pulp_id, path, platforms from rpm_image where distribution_tree_id = '0de942aa-2116-454c-9235-bbdcad17b488';
               pulp_id                |           path            |  platforms  
--------------------------------------+---------------------------+-------------
 1360f187-5e59-4337-8368-2859d3fc83d7 | images/pxeboot/vmlinuz    | x86_64, xen
 99b1e311-4fa3-4661-a7ce-6586366d8b16 | images/pxeboot/initrd.img | x86_64, xen
 5e29d099-ff40-4203-acde-b76442af5dea | images/efiboot.img        | x86_64
 83f2e0f1-6f05-4618-be98-232078df5180 | images/boot.iso           | x86_64
(4 rows)

pulp=> 

I'm going to add "sync to pulp2, and migrate" into the experiment next.

Also - on the Pulp2 side of things - is your CentOS8-B

One last thing - what do you get when you issue

select * from core_contentartifact where content_id = ''0de942aa-2116-454c-9235-bbdcad17b488';

If you've sync'd CentOS8 'on_demand' in Pulp2, the only artifact guaranteed to be there is .treeinfo. On my system, post-migration, as an example:

pulp=> select * from core_contentartifact where content_id = 'fe6d3d49-559a-461a-8348-1fb7ea1e5cc1';
               pulp_id                |         pulp_created          |       pulp_last_updated       |       relative_path       |             artifact_id           
   |              content_id              
--------------------------------------+-------------------------------+-------------------------------+---------------------------+-----------------------------------
---+--------------------------------------
 45710880-a9ff-4ca3-b5c4-674b9a7e3146 | 2021-03-15 18:22:22.314304+00 | 2021-03-15 18:22:22.314315+00 | images/boot.iso           |                                   
   | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
 97249018-ec6c-4f27-abf6-f9b3b08d59df | 2021-03-15 18:22:22.314337+00 | 2021-03-15 18:22:22.314345+00 | images/efiboot.img        |                                   
   | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
 ca09fcf1-6574-4869-85c7-1769994b2d16 | 2021-03-15 18:22:22.314362+00 | 2021-03-15 18:22:22.314368+00 | images/pxeboot/initrd.img |                                   
   | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
 f896ed00-8b07-4c5b-b7d8-754aa7f9c292 | 2021-03-15 18:22:22.314384+00 | 2021-03-15 18:22:22.314391+00 | images/pxeboot/vmlinuz    |                                   
   | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
 3ec58c03-3c03-45eb-a119-a8c872bc89f2 | 2021-03-15 18:22:22.314407+00 | 2021-03-15 18:22:22.314413+00 | images/install.img        |                                   
   | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
 0189d273-0a39-4a2d-baba-f8c39894b5f5 | 2021-03-15 18:22:22.314428+00 | 2021-03-15 18:22:22.314434+00 | .treeinfo                 | 90e18a4a-58f9-4ab7-acb0-5664f6f3ff
82 | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
(6 rows)

pulp=> 

DO you still have, or can you reproduce, the logs from around when this failure happens? Is it pulp complaining, or katello?

Hi, thanks for looking into this problem.

ggainey wrote:

ggainey wrote:

@fbachmann - what does the following show on the pulp3 side of the world?

select pulp_id, path, platforms from rpm_image where distribution_tree_id = '0de942aa-2116-454c-9235-bbdcad17b488';

Looks like what you posted above:

# select pulp_id, path, platforms from rpm_image where distribution_tree_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
               pulp_id                |           path            |  platforms
--------------------------------------+---------------------------+-------------
 134ebee3-5637-4827-9617-e598796d96b3 | images/boot.iso           | x86_64
 e52d6b37-dcec-4532-83ee-2df4b775bdd0 | images/efiboot.img        | x86_64
 74bb8798-1c6e-4976-8d0e-0293fdbd43d8 | images/pxeboot/initrd.img | x86_64, xen
 a5a23885-e830-4433-9ad1-fa3e92c491f6 | images/pxeboot/vmlinuz    | x86_64, xen
(4 rows)

Also - on the Pulp2 side of things - is your CentOS8-Base sync'd immediate or on_demand?

Immediate

One last thing - what do you get when you issue

select * from core_contentartifact where content_id = ''0de942aa-2116-454c-9235-bbdcad17b488';

# select * from core_contentartifact where content_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
               pulp_id                |         pulp_created          |       pulp_last_updated       |       relative_path       |             artifact_id              |              content_id
--------------------------------------+-------------------------------+-------------------------------+---------------------------+--------------------------------------+--------------------------------------
 323d4eb1-3871-4f96-a36a-d11c99499aa3 | 2021-03-04 11:25:54.692017+01 | 2021-03-04 11:25:54.692029+01 | images/boot.iso           | a28bb69d-d01b-41d8-8837-ed59d9e79f60 | 6995fa03-af4e-4f86-b324-a9888309a5e7
 3bde7ca5-91ed-44d0-a77d-f9b0908f940e | 2021-03-04 11:25:54.692061+01 | 2021-03-04 11:25:54.692073+01 | images/efiboot.img        | 4768891c-2c7e-479a-af26-b339556ce05c | 6995fa03-af4e-4f86-b324-a9888309a5e7
 621c1310-df64-40d3-8427-73c59a574686 | 2021-03-04 11:25:54.692193+01 | 2021-03-04 11:25:54.692206+01 | images/install.img        | 9928d035-9614-4940-a9bd-b47215f56c61 | 6995fa03-af4e-4f86-b324-a9888309a5e7
 7dfee54c-6412-42f8-a722-5a132671b36c | 2021-03-04 11:25:54.692105+01 | 2021-03-04 11:25:54.692118+01 | images/pxeboot/initrd.img | c4088184-50a9-480b-b5ec-24c949ddd825 | 6995fa03-af4e-4f86-b324-a9888309a5e7
 f493333d-79fa-49ef-8056-c365ff0b1804 | 2021-03-04 11:25:54.692149+01 | 2021-03-04 11:25:54.692162+01 | images/pxeboot/vmlinuz    | 7526306a-1454-4f11-a9bc-42e17dd6797f | 6995fa03-af4e-4f86-b324-a9888309a5e7
 978d8ce0-7a98-4d76-93ba-3a8cbca96c7b | 2021-03-04 11:25:54.692238+01 | 2021-03-04 11:25:54.69225+01  | .treeinfo                 | 11693a6b-56c6-4371-81e9-97ed753027c8 | 6995fa03-af4e-4f86-b324-a9888309a5e7
(6 rows)

DO you still have, or can you reproduce, the logs from around when this failure happens? Is it pulp complaining, or katello?

I am afraid I do not know enough about the inner workings of either Katello or Pulp to answer this question with confidence.

This is from pulp.log:

Mar  4 11:26:00 foreman pulpcore-api: - - [04/Mar/2021:10:26:00 +0000] "GET /pulp/api/v3/task-groups/5c7decea-f2be-4d4c-9b78-b1d19fc9f875/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
Mar  4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar  4 11:26:05 foreman pulpcore-worker-4: rv = job.perform()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar  4 11:26:05 foreman pulpcore-worker-4: self._result = self._execute()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar  4 11:26:05 foreman pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Mar  4 11:26:05 foreman pulpcore-worker-4: migrate_content(plan, skip_corrupted=skip_corrupted)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Mar  4 11:26:05 foreman pulpcore-worker-4: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Mar  4 11:26:05 foreman pulpcore-worker-4: loop.run_until_complete(dm.create())
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Mar  4 11:26:05 foreman pulpcore-worker-4: return future.result()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 90, in create
Mar  4 11:26:05 foreman pulpcore-worker-4: await pipeline
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Mar  4 11:26:05 foreman pulpcore-worker-4: await asyncio.gather(*futures)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Mar  4 11:26:05 foreman pulpcore-worker-4: await self.run()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Mar  4 11:26:05 foreman pulpcore-worker-4: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Mar  4 11:26:05 foreman pulpcore-worker-4: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Mar  4 11:26:05 foreman pulpcore-worker-4: ValueError: No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=6995fa03-af4e-4f86-b324-a9888309a5e7>"
Mar  4 11:26:05 foreman pulpcore-worker-4: Traceback (most recent call last):
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar  4 11:26:05 foreman pulpcore-worker-4: rv = job.perform()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar  4 11:26:05 foreman pulpcore-worker-4: self._result = self._execute()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar  4 11:26:05 foreman pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Mar  4 11:26:05 foreman pulpcore-worker-4: migrate_content(plan, skip_corrupted=skip_corrupted)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Mar  4 11:26:05 foreman pulpcore-worker-4: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Mar  4 11:26:05 foreman pulpcore-worker-4: loop.run_until_complete(dm.create())
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Mar  4 11:26:05 foreman pulpcore-worker-4: return future.result()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 90, in create
Mar  4 11:26:05 foreman pulpcore-worker-4: await pipeline
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Mar  4 11:26:05 foreman pulpcore-worker-4: await asyncio.gather(*futures)
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Mar  4 11:26:05 foreman pulpcore-worker-4: await self.run()
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Mar  4 11:26:05 foreman pulpcore-worker-4: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Mar  4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Mar  4 11:26:05 foreman pulpcore-worker-4: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Mar  4 11:26:05 foreman pulpcore-worker-4: ValueError: No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=6995fa03-af4e-4f86-b324-a9888309a5e7>"
Mar  4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:INFO: Cleaning registries for queue: 17850@foreman.ubl-is.de
Mar  4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:INFO: 17850@foreman.ubl-is.de: a567276e-d8d8-4f68-994c-a2b05b12e052
Mar  4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:INFO: 17850@foreman.ubl-is.de: Job OK (a567276e-d8d8-4f68-994c-a2b05b12e052)
Mar  4 11:26:16 foreman pulpcore-api: - - [04/Mar/2021:10:26:16 +0000] "GET /pulp/api/v3/tasks/4ad09597-f027-404e-b2e2-4072b41cc2b1/ HTTP/1.1" 200 7314 "-" "OpenAPI-Generator/3.7.1/ruby"

This is from Foreman's production.log:

2021-03-04T11:26:13 [I|app|5d4d3b08] Completed 200 OK in 50ms (Views: 35.0ms | ActiveRecord: 9.0ms | Allocations: 33231)
2021-03-04T11:26:15 [I|app|217664cb] Started GET "/notification_recipients" for 127.0.0.1 at 2021-03-04 11:26:15 +0100
2021-03-04T11:26:15 [I|app|217664cb] Processing by NotificationRecipientsController#index as JSON
2021-03-04T11:26:15 [I|app|217664cb] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 2.3ms | Allocations: 2425)
2021-03-04T11:26:16 [E|bac|] No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=6995fa03-af4e-4f86-b324-a9888309a5e7>" (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-03-04T11:26:16 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 5dfd06d7-4d68-4977-8b2f-63445abc8e25, execution_plan_id: 2a385a9b-4109-4297-8aa9-13db7f126edb} state changed: stopped  result: warning
2021-03-04T11:26:16 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 5dfd06d7-4d68-4977-8b2f-63445abc8e25, execution_plan_id: 2a385a9b-4109-4297-8aa9-13db7f126edb} state changed: stopped  result: warning
2021-03-04T11:26:18 [I|app|bb246384] Started GET "/foreman_tasks/api/tasks/5dfd06d7-4d68-4977-8b2f-63445abc8e25/details?include_permissions" for 127.0.0.1 at 2021-03-04 11:26:18 +0100

Please let me know if you need anything else.

Kind regards Florian

Hm. OK, well this is a lot of data for me to dig into, thanks much! I'll see if I can figure out what's going on here - stay tuned!

@fbachmann I have tried a number of different scenarios to reproduce this, and have had zero luck to date. I'm running on Centos7, pulp2.21.5, pulpcore 3.11dev, pulp_rpm 3.10dev, migration 0.10.0dev. While we've done some work in the area that generates the exception since 3.7, none of it looks like it would address this particular failure.

Is there any chance you could upgrade to pulpcore3.9 or 3.11 and retry? RPMs are available from https://yum.theforeman.org/pulpcore/ . I'm kind of out of ideas at this point :(

Just to get various experiments down in one place - I've tried various combinations of immediate and on_demand on the Pulp2 side, including starting with one, migrating, switching to the other and re-syncing, and then re-migrating. We have

Starting with clean DBs on both sides, here's a set of commands to do the basic sync-immediate-and-migrate:

pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id c8 --feed http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ --download-policy immediate
pulp-admin rpm repo sync run --repo-id c8

pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}'
pulp migration plan run --href HREF-FROM-ABOVE

Between experiments, clean out both sides:

pulp-admin rpm repo delete --repo-id c8
pulp-admin orphan remove --all
pclean  # dev-env alias that drops/recreates Pulp3 db

Replace the "...immediate", above, and use this line to test on_demand:

pulp-admin rpm repo create --repo-id c8 --feed http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ --download-policy on_demand

Do the first experiment, then switch Pulp2 from immediate to on_demand, resync and redo the migration:

pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id c8 --feed http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ --download-policy immediate
pulp-admin rpm repo sync run --repo-id c8

pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}'
pulp migration plan run --href HREF-FROM-ABOVE

pulp-admin rpm repo update --repo-id c8 --download-policy on_demand
pulp-admin rpm repo sync run  --repo-id c8 
pulp migration plan run --href HREF-FROM-ABOVE

Repeat the last test, starting with on_demand and switching to immediate.

Digging into the CentOS remote-repo, everything looks fine at http://mirror.centos.org/centos/8/BaseOS/x86_64/os/. It was last updated on 16-MAR, and OP didn't hit the problem until 22-MAR, so it's un/less-likely that there was some oddness in the remote causing us trouble.

OP db queries on the Pulp3 side, show the data I would expect for a distribution-tree.

Hopefully we can get some more eyes on the above, that can spot a hole I have missed.

ggainey, I have one unconfirmed theory, a bit painful one, so bear with me.

What if... it's the AppStream kickstart repo interfering with the BaseOS one?
If both are present in pulp2 and one of them is migrated first...
They both would be treated as the same distribution tree I think.
I still can't fully explain how to get to what OP sees but my first instinct is to test those two.

See the CentOS8 treeinfos
http://mirror.centos.org/centos/8/BaseOS/x86_64/os/.treeinfo
http://mirror.centos.org/centos/8/AppStream/x86_64/kickstart/.treeinfo
And uniqueness constraint for a distribution tree https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/distribution.py#L133-L140
And now I'll tell you that at migration time build_timestamp is set to 0 to let Pulp 3 bring in addons on the upcoming sync.
(Maybe it's better to do build_timestamp minus 1, instead of setting it to zero?)

Not sure if it's the reason for this specific ticket but maybe still worth testing.

ttereshc wrote:

ggainey, I have one unconfirmed theory, a bit painful one, so bear with me.

That's a great idea!

I have access to an empty 3.7 system. While I'm getting it set up, and can run this test in my dev-env, maybe it'll uncover something.

More eyes is always better!

Aaaand ttereshc wins the prize!

If you have CentOS8-Base and CentOs8-AppStream-Kickstart sync'd, and you migrate them, you trigger this failure. I've recreated this in our dev-env with current-master, which makes debugging MUCH MUCH easier.

@fbachmann - there is Hope! Also, I suspect that removing the 8-AppStream-Kickstart tree from your Pulp2 would be a workaround, if you can't wait.

ggainey, great news. Please consider testing RHEL8 repos as well, since the treeinfo files might slightly differ. It would be good to understand if it's CentOS only, or RHEL as well. Thanks for working on it!

commit 072677 in https://github.com/pulp/pulpcore-plugin introduced an optimization that may be faster, but is (in this specific case) wrong. AppStream has a .treeinfo, but no artifacts. The prefetch logic added in 072677 results in code that attempts to find the remote, for artifacts associated with the BaseOS tree, which AppStream doesn't know about, and we throw the error.

See #4404 for details on the (substantial) performance impact. Investigation into the right way to address this, without losing the performance-gain, continues.