Project

Profile

Help

Issue #7681

ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree) ..."

Added by dalley 12 months ago. Updated 8 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 89
Quarter:

Description

  1. Sync centos8-baseos repo into Pulp 2
  2. Migrate it into Pulp 3
  3. Sync centos8-kickstart repo into Pulp 2
  4. Migrate again using the same plan as before (migrate content but not centos8-kickstart repository)

Result:

Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: rv = job.perform()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: self._result = self._execute()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return self.func(*self.args, **self.kwargs)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: migrate_content(plan)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: plugin.migrator.migrate_content_to_pulp3()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: loop.run_until_complete(dm.create())
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return future.result()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await asyncio.gather(*futures)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await self.run()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree): pk=1dddfb60-f80f-4842-a791-2
803f6e4751a>"
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: Traceback (most recent call last):
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: rv = job.perform()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: self._result = self._execute()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return self.func(*self.args, **self.kwargs)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: migrate_content(plan)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: plugin.migrator.migrate_content_to_pulp3()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: loop.run_until_complete(dm.create())
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return future.result()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await asyncio.gather(*futures)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await self.run()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree): pk=1dddfb60-f80f-4842-a791-2
803f6e4751a>"

Possibly related to https://github.com/pulp/pulp-2to3-migration/pull/240/ ?

Also possibly related to https://pulp.plan.io/issues/6944

If the repository is part of the migration plan, it successfully migrates. Also if both repositories are available during the first migration it successfully migrates.

7681_no_declared_artifact.sh (1.52 KB) 7681_no_declared_artifact.sh dalley, 11/12/2020 04:04 AM

Associated revisions

Revision 989cbe23 View on GitHub
Added by dalley 8 months ago

Skip multi-artifact content properly when an artifact is unavailable.

closes: #7681 https://pulp.plan.io/issues/7681

Revision 989cbe23 View on GitHub
Added by dalley 8 months ago

Skip multi-artifact content properly when an artifact is unavailable.

closes: #7681 https://pulp.plan.io/issues/7681

Revision 989cbe23 View on GitHub
Added by dalley 8 months ago

Skip multi-artifact content properly when an artifact is unavailable.

closes: #7681 https://pulp.plan.io/issues/7681

History

#1 Updated by dalley 12 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#2 Updated by dalley 12 months ago

  • Description updated (diff)

#3 Updated by ttereshc 12 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 83

#4 Updated by rchan 12 months ago

  • Sprint changed from Sprint 83 to Sprint 84

#5 Updated by rchan 11 months ago

  • Sprint changed from Sprint 84 to Sprint 85

#6 Updated by dalley 11 months ago

Reproducer script attached

#7 Updated by ttereshc 11 months ago

  • Tags Katello added

#8 Updated by rchan 11 months ago

  • Sprint changed from Sprint 85 to Sprint 86

#9 Updated by rchan 10 months ago

  • Sprint changed from Sprint 86 to Sprint 87

#10 Updated by rchan 9 months ago

  • Sprint changed from Sprint 87 to Sprint 88

#11 Updated by ttereshc 8 months ago

  • Sprint/Milestone set to 0.9.0

#12 Updated by dalley 8 months ago

It looks like the .treeinfo declarative artifact is attached to a declarative content where the contentartifact relative path is images/boot.iso

#13 Updated by rchan 8 months ago

  • Sprint changed from Sprint 88 to Sprint 89

#14 Updated by dalley 8 months ago

        for d_content in batch:
            for content_artifact in d_content.content._remote_artifact_saver_cas:   #  <--- content_artifact.relative_path == 'images/boot.iso'
                for d_artifact in d_content.d_artifacts:  # d_content.d_artifacts == [ {{ca: relative_path=='.treeinfo'}} ]    # only one item, no match, "else" is triggered
                    if d_artifact.relative_path == content_artifact.relative_path:
                        break
                else:
                    msg = _('No declared artifact with relative path "{rp}" for content "{c}"')
                    raise ValueError(
                        msg.format(rp=content_artifact.relative_path, c=d_content.content)
                    )
                for remote_artifact in content_artifact._remote_artifact_saver_ras:
                    if remote_artifact.remote_id == d_artifact.remote.pk:
                        break
                else:
                    if d_artifact.remote:
                        remote_artifact = self._create_remote_artifact(d_artifact, content_artifact)
                        needed_ras.append(remote_artifact)
        return needed_ras
[c.relative_path for c in ContentArtifact.objects.filter(content=d_content.content)]
Out[9]: 
['images/boot.iso',
 'images/efiboot.img',
 'images/install.img',
 'images/pxeboot/initrd.img',
 'images/pxeboot/vmlinuz',
 '.treeinfo']
[c.relative_path for c in d_content.content._remote_artifact_saver_cas]
Out[10]: 
['images/boot.iso',
 'images/efiboot.img',
 'images/pxeboot/initrd.img',
 'images/pxeboot/vmlinuz',
 'images/install.img',
 '.treeinfo']
[d.relative_path for d in d_content.d_artifacts]
Out[11]: ['.treeinfo']

From this info, it looks like the problem is that there are several content artifacts that don't have matching items in d_artifacts

#15 Updated by dalley 8 months ago

Which happens because both "downloaded" and "remote" are False, so it skips over the artifacts with "cannot download" warning.

Which happens because the Pulp2Importer isn't available because it wasn't in the migration plan.

So the "real" problem is that it's not skipping the entire DistributionTree content unit when one of the artifacts can't be found, it's just skipping the artifacts.

#16 Updated by dalley 8 months ago

  • Status changed from ASSIGNED to POST

#17 Updated by dalley 8 months ago

  • Status changed from POST to MODIFIED

#18 Updated by ttereshc 8 months ago

  • Sprint/Milestone changed from 0.9.0 to 0.7.0

#19 Updated by pulpbot 8 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF