Issue #7681: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree) ..." - Migration Plugin - Pulp

Actions

Send by e-mail Copy link

Issue #7681

closed

ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree) ..."

Added by dalley about 4 years ago. Updated almost 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

dalley

Sprint/Milestone:

0.7.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Sprint 89

Quarter:

Description

Sync centos8-baseos repo into Pulp 2
Migrate it into Pulp 3
Sync centos8-kickstart repo into Pulp 2
Migrate again using the same plan as before (migrate content but not centos8-kickstart repository)

Result:

Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: rv = job.perform()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: self._result = self._execute()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return self.func(*self.args, **self.kwargs)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: migrate_content(plan)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: plugin.migrator.migrate_content_to_pulp3()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: loop.run_until_complete(dm.create())
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return future.result()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await asyncio.gather(*futures)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await self.run()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree): pk=1dddfb60-f80f-4842-a791-2
803f6e4751a>"
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: Traceback (most recent call last):
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: rv = job.perform()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: self._result = self._execute()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return self.func(*self.args, **self.kwargs)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: migrate_content(plan)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: plugin.migrator.migrate_content_to_pulp3()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: loop.run_until_complete(dm.create())
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return future.result()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await asyncio.gather(*futures)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await self.run()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree): pk=1dddfb60-f80f-4842-a791-2
803f6e4751a>"

Possibly related to https://github.com/pulp/pulp-2to3-migration/pull/240/ ?

Also possibly related to https://pulp.plan.io/issues/6944

If the repository is part of the migration plan, it successfully migrates. Also if both repositories are available during the first migration it successfully migrates.

Files

7681_no_declared_artifact.sh (1.52 KB) 7681_no_declared_artifact.sh

dalley, 11/12/2020 04:04 AM

Actions

Copy link

Updated by dalley about 4 years ago

Status changed from NEW to ASSIGNED
Assignee set to dalley

Actions

Copy link

Updated by dalley about 4 years ago

Description updated (diff)

Actions

Copy link

Updated by ttereshc about 4 years ago

Triaged changed from No to Yes
Sprint set to Sprint 83

Actions

Copy link

Updated by rchan about 4 years ago

Sprint changed from Sprint 83 to Sprint 84

Actions

Copy link

Updated by rchan about 4 years ago

Sprint changed from Sprint 84 to Sprint 85

Actions

Copy link

Updated by dalley about 4 years ago

File 7681_no_declared_artifact.sh 7681_no_declared_artifact.sh added

Reproducer script attached

Actions

Copy link

Updated by ttereshc about 4 years ago

Tags Katello added

Actions

Copy link

Updated by rchan about 4 years ago

Sprint changed from Sprint 85 to Sprint 86

Actions

Copy link

Updated by rchan about 4 years ago

Sprint changed from Sprint 86 to Sprint 87

Actions

Copy link

#10

Updated by rchan about 4 years ago

Sprint changed from Sprint 87 to Sprint 88

Actions

Copy link

#11

Updated by ttereshc almost 4 years ago

Sprint/Milestone set to 0.9.0

Actions

Copy link

#12

Updated by dalley almost 4 years ago

It looks like the .treeinfo declarative artifact is attached to a declarative content where the contentartifact relative path is images/boot.iso

Actions

Copy link

#13

Updated by rchan almost 4 years ago

Sprint changed from Sprint 88 to Sprint 89

Actions

Copy link

#14

Updated by dalley almost 4 years ago

        for d_content in batch:
            for content_artifact in d_content.content._remote_artifact_saver_cas:   #  <--- content_artifact.relative_path == 'images/boot.iso'
                for d_artifact in d_content.d_artifacts:  # d_content.d_artifacts == [ {{ca: relative_path=='.treeinfo'}} ]    # only one item, no match, "else" is triggered
                    if d_artifact.relative_path == content_artifact.relative_path:
                        break
                else:
                    msg = _('No declared artifact with relative path "{rp}" for content "{c}"')
                    raise ValueError(
                        msg.format(rp=content_artifact.relative_path, c=d_content.content)
                    )
                for remote_artifact in content_artifact._remote_artifact_saver_ras:
                    if remote_artifact.remote_id == d_artifact.remote.pk:
                        break
                else:
                    if d_artifact.remote:
                        remote_artifact = self._create_remote_artifact(d_artifact, content_artifact)
                        needed_ras.append(remote_artifact)
        return needed_ras

[c.relative_path for c in ContentArtifact.objects.filter(content=d_content.content)]
Out[9]: 
['images/boot.iso',
 'images/efiboot.img',
 'images/install.img',
 'images/pxeboot/initrd.img',
 'images/pxeboot/vmlinuz',
 '.treeinfo']
[c.relative_path for c in d_content.content._remote_artifact_saver_cas]
Out[10]: 
['images/boot.iso',
 'images/efiboot.img',
 'images/pxeboot/initrd.img',
 'images/pxeboot/vmlinuz',
 'images/install.img',
 '.treeinfo']
[d.relative_path for d in d_content.d_artifacts]
Out[11]: ['.treeinfo']

From this info, it looks like the problem is that there are several content artifacts that don't have matching items in d_artifacts

Actions

Copy link

#15

Updated by dalley almost 4 years ago

Which happens because both "downloaded" and "remote" are False, so it skips over the artifacts with "cannot download" warning.

Which happens because the Pulp2Importer isn't available because it wasn't in the migration plan.

So the "real" problem is that it's not skipping the entire DistributionTree content unit when one of the artifacts can't be found, it's just skipping the artifacts.

Actions

Copy link

#16