https://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-03-10T16:29:38ZPulpPulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=684942021-03-10T16:29:38Ziballou
<ul></ul><p>Hi Florian, did you mean "<a href="http://mirror.centos.org/centos/8/BaseOS/x86_64/os/" class="external">http://mirror.centos.org/centos/8/BaseOS/x86_64/os/</a>" for the repo URL?</p>
<p>Also here's the link to the Foreman community forum thread that preceded this bug: <a href="https://community.theforeman.org/t/katello-3-18-1-content-migration-to-pulp-3-fails-with-no-declared-artifact-message/22645/2" class="external">https://community.theforeman.org/t/katello-3-18-1-content-migration-to-pulp-3-fails-with-no-declared-artifact-message/22645/2</a></p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=684952021-03-10T16:45:09Zfbachmann
<ul></ul><p>Oops, I indeed meant to write "<a href="http://mirror.centos.org/centos/8/BaseOS/x86_64/os/" class="external">http://mirror.centos.org/centos/8/BaseOS/x86_64/os/</a>" for the repo URL. Sorry...</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=685772021-03-12T16:09:40Zfao89
<ul><li><strong>Project</strong> changed from <i>Pulp</i> to <i>Migration Plugin</i></li><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li><li><strong>Sprint</strong> set to <i>Sprint 92</i></li></ul> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=686242021-03-15T18:03:36Zggainey
<ul></ul><p>@fbachmann - what does the following show on the pulp3 side of the world?</p>
<pre><code>select pulp_id, path, platforms from rpm_image where distribution_tree_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
</code></pre>
<p>On my pulp3 instance after sync'ing CentOS8-Base, I see the following:</p>
<pre><code>pulp=> select pulp_id, path, platforms from rpm_image where distribution_tree_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
pulp_id | path | platforms
--------------------------------------+---------------------------+-------------
1360f187-5e59-4337-8368-2859d3fc83d7 | images/pxeboot/vmlinuz | x86_64, xen
99b1e311-4fa3-4661-a7ce-6586366d8b16 | images/pxeboot/initrd.img | x86_64, xen
5e29d099-ff40-4203-acde-b76442af5dea | images/efiboot.img | x86_64
83f2e0f1-6f05-4618-be98-232078df5180 | images/boot.iso | x86_64
(4 rows)
pulp=>
</code></pre>
<p>I'm going to add "sync to pulp2, and migrate" into the experiment next.</p>
<p>Also - on the Pulp2 side of things - is your CentOS8-Base sync'd immediate or on_demand?</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=686322021-03-15T18:34:01Zggainey
<ul></ul><p>ggainey wrote:</p>
<blockquote>
<p>@fbachmann - what does the following show on the pulp3 side of the world?</p>
<pre><code>select pulp_id, path, platforms from rpm_image where distribution_tree_id = '0de942aa-2116-454c-9235-bbdcad17b488';
</code></pre>
<p>On my pulp3 instance after sync'ing CentOS8-Base, I see the following:</p>
<pre><code>pulp=> select pulp_id, path, platforms from rpm_image where distribution_tree_id = '0de942aa-2116-454c-9235-bbdcad17b488';
pulp_id | path | platforms
--------------------------------------+---------------------------+-------------
1360f187-5e59-4337-8368-2859d3fc83d7 | images/pxeboot/vmlinuz | x86_64, xen
99b1e311-4fa3-4661-a7ce-6586366d8b16 | images/pxeboot/initrd.img | x86_64, xen
5e29d099-ff40-4203-acde-b76442af5dea | images/efiboot.img | x86_64
83f2e0f1-6f05-4618-be98-232078df5180 | images/boot.iso | x86_64
(4 rows)
pulp=>
</code></pre>
<p>I'm going to add "sync to pulp2, and migrate" into the experiment next.</p>
<p>Also - on the Pulp2 side of things - is your CentOS8-B</p>
</blockquote>
<p>One last thing - what do you get when you issue</p>
<pre><code>select * from core_contentartifact where content_id = ''0de942aa-2116-454c-9235-bbdcad17b488';
</code></pre>
<p>If you've sync'd CentOS8 'on_demand' in Pulp2, the only artifact guaranteed to be there is .treeinfo. On my system, post-migration, as an example:</p>
<pre><code>pulp=> select * from core_contentartifact where content_id = 'fe6d3d49-559a-461a-8348-1fb7ea1e5cc1';
pulp_id | pulp_created | pulp_last_updated | relative_path | artifact_id
| content_id
--------------------------------------+-------------------------------+-------------------------------+---------------------------+-----------------------------------
---+--------------------------------------
45710880-a9ff-4ca3-b5c4-674b9a7e3146 | 2021-03-15 18:22:22.314304+00 | 2021-03-15 18:22:22.314315+00 | images/boot.iso |
| fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
97249018-ec6c-4f27-abf6-f9b3b08d59df | 2021-03-15 18:22:22.314337+00 | 2021-03-15 18:22:22.314345+00 | images/efiboot.img |
| fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
ca09fcf1-6574-4869-85c7-1769994b2d16 | 2021-03-15 18:22:22.314362+00 | 2021-03-15 18:22:22.314368+00 | images/pxeboot/initrd.img |
| fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
f896ed00-8b07-4c5b-b7d8-754aa7f9c292 | 2021-03-15 18:22:22.314384+00 | 2021-03-15 18:22:22.314391+00 | images/pxeboot/vmlinuz |
| fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
3ec58c03-3c03-45eb-a119-a8c872bc89f2 | 2021-03-15 18:22:22.314407+00 | 2021-03-15 18:22:22.314413+00 | images/install.img |
| fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
0189d273-0a39-4a2d-baba-f8c39894b5f5 | 2021-03-15 18:22:22.314428+00 | 2021-03-15 18:22:22.314434+00 | .treeinfo | 90e18a4a-58f9-4ab7-acb0-5664f6f3ff
82 | fe6d3d49-559a-461a-8348-1fb7ea1e5cc1
(6 rows)
pulp=>
</code></pre>
<p>DO you still have, or can you reproduce, the logs from around when this failure happens? Is it pulp complaining, or katello?</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=686602021-03-15T21:02:43Zfbachmann
<ul></ul><p>Hi, thanks for looking into this problem.</p>
<p>ggainey wrote:</p>
<blockquote>
<p>ggainey wrote:</p>
<blockquote>
<p>@fbachmann - what does the following show on the pulp3 side of the world?</p>
<pre><code>select pulp_id, path, platforms from rpm_image where distribution_tree_id = '0de942aa-2116-454c-9235-bbdcad17b488';
</code></pre>
</blockquote>
</blockquote>
<p>Looks like what you posted above:</p>
<pre><code># select pulp_id, path, platforms from rpm_image where distribution_tree_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
pulp_id | path | platforms
--------------------------------------+---------------------------+-------------
134ebee3-5637-4827-9617-e598796d96b3 | images/boot.iso | x86_64
e52d6b37-dcec-4532-83ee-2df4b775bdd0 | images/efiboot.img | x86_64
74bb8798-1c6e-4976-8d0e-0293fdbd43d8 | images/pxeboot/initrd.img | x86_64, xen
a5a23885-e830-4433-9ad1-fa3e92c491f6 | images/pxeboot/vmlinuz | x86_64, xen
(4 rows)
</code></pre>
<blockquote>
<blockquote>
<p>Also - on the Pulp2 side of things - is your CentOS8-Base sync'd immediate or on_demand?</p>
</blockquote>
</blockquote>
<p>Immediate</p>
<blockquote>
<p>One last thing - what do you get when you issue</p>
<pre><code>select * from core_contentartifact where content_id = ''0de942aa-2116-454c-9235-bbdcad17b488';
</code></pre>
</blockquote>
<pre><code># select * from core_contentartifact where content_id = '6995fa03-af4e-4f86-b324-a9888309a5e7';
pulp_id | pulp_created | pulp_last_updated | relative_path | artifact_id | content_id
--------------------------------------+-------------------------------+-------------------------------+---------------------------+--------------------------------------+--------------------------------------
323d4eb1-3871-4f96-a36a-d11c99499aa3 | 2021-03-04 11:25:54.692017+01 | 2021-03-04 11:25:54.692029+01 | images/boot.iso | a28bb69d-d01b-41d8-8837-ed59d9e79f60 | 6995fa03-af4e-4f86-b324-a9888309a5e7
3bde7ca5-91ed-44d0-a77d-f9b0908f940e | 2021-03-04 11:25:54.692061+01 | 2021-03-04 11:25:54.692073+01 | images/efiboot.img | 4768891c-2c7e-479a-af26-b339556ce05c | 6995fa03-af4e-4f86-b324-a9888309a5e7
621c1310-df64-40d3-8427-73c59a574686 | 2021-03-04 11:25:54.692193+01 | 2021-03-04 11:25:54.692206+01 | images/install.img | 9928d035-9614-4940-a9bd-b47215f56c61 | 6995fa03-af4e-4f86-b324-a9888309a5e7
7dfee54c-6412-42f8-a722-5a132671b36c | 2021-03-04 11:25:54.692105+01 | 2021-03-04 11:25:54.692118+01 | images/pxeboot/initrd.img | c4088184-50a9-480b-b5ec-24c949ddd825 | 6995fa03-af4e-4f86-b324-a9888309a5e7
f493333d-79fa-49ef-8056-c365ff0b1804 | 2021-03-04 11:25:54.692149+01 | 2021-03-04 11:25:54.692162+01 | images/pxeboot/vmlinuz | 7526306a-1454-4f11-a9bc-42e17dd6797f | 6995fa03-af4e-4f86-b324-a9888309a5e7
978d8ce0-7a98-4d76-93ba-3a8cbca96c7b | 2021-03-04 11:25:54.692238+01 | 2021-03-04 11:25:54.69225+01 | .treeinfo | 11693a6b-56c6-4371-81e9-97ed753027c8 | 6995fa03-af4e-4f86-b324-a9888309a5e7
(6 rows)
</code></pre>
<blockquote>
<p>DO you still have, or can you reproduce, the logs from around when this failure happens? Is it pulp complaining, or katello?</p>
</blockquote>
<p>I am afraid I do not know enough about the inner workings of either Katello or Pulp to answer this question with confidence.</p>
<p>This is from <code>pulp.log</code>:</p>
<pre><code>Mar 4 11:26:00 foreman pulpcore-api: - - [04/Mar/2021:10:26:00 +0000] "GET /pulp/api/v3/task-groups/5c7decea-f2be-4d4c-9b78-b1d19fc9f875/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
Mar 4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar 4 11:26:05 foreman pulpcore-worker-4: rv = job.perform()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar 4 11:26:05 foreman pulpcore-worker-4: self._result = self._execute()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar 4 11:26:05 foreman pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Mar 4 11:26:05 foreman pulpcore-worker-4: migrate_content(plan, skip_corrupted=skip_corrupted)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Mar 4 11:26:05 foreman pulpcore-worker-4: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Mar 4 11:26:05 foreman pulpcore-worker-4: loop.run_until_complete(dm.create())
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Mar 4 11:26:05 foreman pulpcore-worker-4: return future.result()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 90, in create
Mar 4 11:26:05 foreman pulpcore-worker-4: await pipeline
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Mar 4 11:26:05 foreman pulpcore-worker-4: await asyncio.gather(*futures)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Mar 4 11:26:05 foreman pulpcore-worker-4: await self.run()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Mar 4 11:26:05 foreman pulpcore-worker-4: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Mar 4 11:26:05 foreman pulpcore-worker-4: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Mar 4 11:26:05 foreman pulpcore-worker-4: ValueError: No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=6995fa03-af4e-4f86-b324-a9888309a5e7>"
Mar 4 11:26:05 foreman pulpcore-worker-4: Traceback (most recent call last):
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar 4 11:26:05 foreman pulpcore-worker-4: rv = job.perform()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar 4 11:26:05 foreman pulpcore-worker-4: self._result = self._execute()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar 4 11:26:05 foreman pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Mar 4 11:26:05 foreman pulpcore-worker-4: migrate_content(plan, skip_corrupted=skip_corrupted)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Mar 4 11:26:05 foreman pulpcore-worker-4: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Mar 4 11:26:05 foreman pulpcore-worker-4: loop.run_until_complete(dm.create())
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Mar 4 11:26:05 foreman pulpcore-worker-4: return future.result()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 90, in create
Mar 4 11:26:05 foreman pulpcore-worker-4: await pipeline
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Mar 4 11:26:05 foreman pulpcore-worker-4: await asyncio.gather(*futures)
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Mar 4 11:26:05 foreman pulpcore-worker-4: await self.run()
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Mar 4 11:26:05 foreman pulpcore-worker-4: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Mar 4 11:26:05 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Mar 4 11:26:05 foreman pulpcore-worker-4: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Mar 4 11:26:05 foreman pulpcore-worker-4: ValueError: No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=6995fa03-af4e-4f86-b324-a9888309a5e7>"
Mar 4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:INFO: Cleaning registries for queue: 17850@foreman.ubl-is.de
Mar 4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:INFO: 17850@foreman.ubl-is.de: a567276e-d8d8-4f68-994c-a2b05b12e052
Mar 4 11:26:05 foreman pulpcore-worker-4: pulp: rq.worker:INFO: 17850@foreman.ubl-is.de: Job OK (a567276e-d8d8-4f68-994c-a2b05b12e052)
Mar 4 11:26:16 foreman pulpcore-api: - - [04/Mar/2021:10:26:16 +0000] "GET /pulp/api/v3/tasks/4ad09597-f027-404e-b2e2-4072b41cc2b1/ HTTP/1.1" 200 7314 "-" "OpenAPI-Generator/3.7.1/ruby"
</code></pre>
<p>This is from Foreman's <code>production.log</code>:</p>
<pre><code>2021-03-04T11:26:13 [I|app|5d4d3b08] Completed 200 OK in 50ms (Views: 35.0ms | ActiveRecord: 9.0ms | Allocations: 33231)
2021-03-04T11:26:15 [I|app|217664cb] Started GET "/notification_recipients" for 127.0.0.1 at 2021-03-04 11:26:15 +0100
2021-03-04T11:26:15 [I|app|217664cb] Processing by NotificationRecipientsController#index as JSON
2021-03-04T11:26:15 [I|app|217664cb] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 2.3ms | Allocations: 2425)
2021-03-04T11:26:16 [E|bac|] No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=6995fa03-af4e-4f86-b324-a9888309a5e7>" (Katello::Errors::Pulp3Error)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
| [ sidekiq ]
| [ concurrent-ruby ]
2021-03-04T11:26:16 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 5dfd06d7-4d68-4977-8b2f-63445abc8e25, execution_plan_id: 2a385a9b-4109-4297-8aa9-13db7f126edb} state changed: stopped result: warning
2021-03-04T11:26:16 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 5dfd06d7-4d68-4977-8b2f-63445abc8e25, execution_plan_id: 2a385a9b-4109-4297-8aa9-13db7f126edb} state changed: stopped result: warning
2021-03-04T11:26:18 [I|app|bb246384] Started GET "/foreman_tasks/api/tasks/5dfd06d7-4d68-4977-8b2f-63445abc8e25/details?include_permissions" for 127.0.0.1 at 2021-03-04 11:26:18 +0100
</code></pre>
<p>Please let me know if you need anything else.</p>
<p>Kind regards
Florian</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=687462021-03-16T20:42:26Zggainey
<ul></ul><p>Hm. OK, well this is a lot of data for me to dig into, thanks much! I'll see if I can figure out what's going on here - stay tuned!</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=689092021-03-19T20:25:42Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 92</i> to <i>Sprint 93</i></li></ul> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=689642021-03-23T16:48:26Zggainey
<ul></ul><p>@fbachmann I have tried a number of different scenarios to reproduce this, and have had zero luck to date. I'm running on Centos7, pulp2.21.5, pulpcore 3.11dev, pulp_rpm 3.10dev, migration 0.10.0dev. While we've done some work in the area that generates the exception since 3.7, none of it looks like it would address this particular failure.</p>
<p>Is there any chance you could upgrade to pulpcore3.9 or 3.11 and retry? RPMs are available from <a href="https://yum.theforeman.org/pulpcore/" class="external">https://yum.theforeman.org/pulpcore/</a> . I'm kind of out of ideas at this point :(</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=690092021-03-24T15:42:45Zggainey
<ul></ul><p>Just to get various experiments down in one place - I've tried various combinations of immediate and on_demand on the Pulp2 side, including starting with one, migrating, switching to the other and re-syncing, and then re-migrating. We have</p>
<p>Starting with clean DBs on both sides, here's a set of commands to do the basic sync-immediate-and-migrate:</p>
<pre><code>pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id c8 --feed http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ --download-policy immediate
pulp-admin rpm repo sync run --repo-id c8
pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}'
pulp migration plan run --href HREF-FROM-ABOVE
</code></pre>
<p>Between experiments, clean out both sides:</p>
<pre><code>pulp-admin rpm repo delete --repo-id c8
pulp-admin orphan remove --all
pclean # dev-env alias that drops/recreates Pulp3 db
</code></pre>
<p>Replace the "...immediate", above, and use this line to test on_demand:</p>
<pre><code>pulp-admin rpm repo create --repo-id c8 --feed http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ --download-policy on_demand
</code></pre>
<p>Do the first experiment, then switch Pulp2 from immediate to on_demand, resync and redo the migration:</p>
<pre><code>pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id c8 --feed http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ --download-policy immediate
pulp-admin rpm repo sync run --repo-id c8
pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}'
pulp migration plan run --href HREF-FROM-ABOVE
pulp-admin rpm repo update --repo-id c8 --download-policy on_demand
pulp-admin rpm repo sync run --repo-id c8
pulp migration plan run --href HREF-FROM-ABOVE
</code></pre>
<p>Repeat the last test, starting with on_demand and switching to immediate.</p>
<p>Digging into the CentOS remote-repo, everything looks fine at <code>http://mirror.centos.org/centos/8/BaseOS/x86_64/os/</code>. It was last updated on 16-MAR, and OP didn't hit the problem until 22-MAR, so it's un/less-likely that there was some oddness in the remote causing us trouble.</p>
<p>OP db queries on the Pulp3 side, show the data I would expect for a distribution-tree.</p>
<p>Hopefully we can get some more eyes on the above, that can spot a hole I have missed.</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=690212021-03-24T18:48:15Zttereshcttereshc@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/14796">ggainey</a>, I have one unconfirmed theory, a bit painful one, so bear with me.</p>
<p>What if... it's the AppStream kickstart repo interfering with the BaseOS one?<br>
If both are present in pulp2 and one of them is migrated first...<br>
They both would be treated as the same distribution tree I think.<br>
I still can't fully explain how to get to what OP sees but my first instinct is to test those two.</p>
<p>See the CentOS8 treeinfos<br>
<a href="http://mirror.centos.org/centos/8/BaseOS/x86_64/os/.treeinfo" class="external">http://mirror.centos.org/centos/8/BaseOS/x86_64/os/.treeinfo</a><br>
<a href="http://mirror.centos.org/centos/8/AppStream/x86_64/kickstart/.treeinfo" class="external">http://mirror.centos.org/centos/8/AppStream/x86_64/kickstart/.treeinfo</a><br>
And uniqueness constraint for a distribution tree <a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/distribution.py#L133-L140" class="external">https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/distribution.py#L133-L140</a><br>
And now I'll tell you that at migration time build_timestamp is set to 0 to let Pulp 3 bring in addons on the upcoming sync.<br>
(Maybe it's better to do build_timestamp minus 1, instead of setting it to zero?)</p>
<p>Not sure if it's the reason for this specific ticket but maybe still worth testing.</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=690222021-03-24T18:51:01Zggainey
<ul></ul><p>ttereshc wrote:</p>
<blockquote>
<p><a class="user active" href="https://pulp.plan.io/users/14796">ggainey</a>, I have one unconfirmed theory, a bit painful one, so bear with me.</p>
</blockquote>
<p>That's a great idea!</p>
<p>I have access to an empty 3.7 system. While I'm getting it set up, and can run this test in my dev-env, maybe it'll uncover something.</p>
<p>More eyes is always better!</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=690232021-03-24T20:00:45Zggainey
<ul></ul><p>Aaaand ttereshc wins the prize!</p>
<p>If you have CentOS8-Base <strong>and</strong> CentOs8-AppStream-Kickstart sync'd, and you migrate them, you trigger this failure. I've recreated this in our dev-env with current-master, which makes debugging MUCH MUCH easier.</p>
<p>@fbachmann - there is Hope! Also, I suspect that removing the 8-AppStream-Kickstart tree from your Pulp2 would be a workaround, if you can't wait.</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=690242021-03-24T20:20:07Zttereshcttereshc@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/14796">ggainey</a>, great news.
Please consider testing RHEL8 repos as well, since the treeinfo files might slightly differ.
It would be good to understand if it's CentOS only, or RHEL as well.
Thanks for working on it!</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=691052021-03-28T15:05:56Zggainey
<ul></ul><p><a href="https://github.com/pulp/pulpcore-plugin/commit/072677" class="external">commit 072677</a> in <a href="https://github.com/pulp/pulpcore-plugin" class="external">https://github.com/pulp/pulpcore-plugin</a> introduced an optimization that may be faster, but is (in this specific case) wrong. AppStream has a .treeinfo, but no artifacts. The prefetch logic added in 072677 results in code that attempts to find the remote, for artifacts associated with the <strong>BaseOS</strong> tree, which AppStream doesn't know about, and we throw the error.</p>
<p>See <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: Sync performance degradation with RemoteArtifactSaver stage (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4404">#4404</a> for details on the (substantial) performance impact. Investigation into the right way to address this, without losing the performance-gain, continues.</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=692562021-03-30T21:18:56Zggainey
<ul><li><strong>Project</strong> changed from <i>Migration Plugin</i> to <i>Pulp</i></li></ul> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=692582021-03-30T21:25:05Zpulpbot
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>POST</i></li></ul><p>PR: <a href="https://github.com/pulp/pulpcore/pull/1219" class="external">https://github.com/pulp/pulpcore/pull/1219</a></p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=692682021-03-31T14:18:19Zggainey
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="Fixed artifact_stages edge case for multi-artifact/multi-remote batches. Only encountered when 2..." href="https://pulp.plan.io/projects/pulp/repository/pulpcore/revisions/48b365d048de15cc922f69b7c64cb0d8fcf20535">pulpcore|48b365d048de15cc922f69b7c64cb0d8fcf20535</a>.</p> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=694642021-04-08T14:39:46Zmdellweg
<ul><li><strong>Sprint/Milestone</strong> set to <i>3.12.0</i></li></ul> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=695302021-04-08T20:56:51Zpulpbot
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul> Pulp - Issue #8377: Content Migration to Pulp 3 with Katello fails with message "No declared artifact"https://pulp.plan.io/issues/8377?journal_id=695732021-04-09T14:18:56Zggainey
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-12 priority-6 priority-default closed" href="/issues/8514">Issue #8514</a>: Foreman Katello: Unable to migrate pulp2 to pulp3</i> added</li></ul>