Project

Profile

Help

Issue #7681

closed

ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree) ..."

Added by dalley over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 89
Quarter:

Description

  1. Sync centos8-baseos repo into Pulp 2
  2. Migrate it into Pulp 3
  3. Sync centos8-kickstart repo into Pulp 2
  4. Migrate again using the same plan as before (migrate content but not centos8-kickstart repository)

Result:

Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: rv = job.perform()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: self._result = self._execute()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return self.func(*self.args, **self.kwargs)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: migrate_content(plan)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: plugin.migrator.migrate_content_to_pulp3()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: loop.run_until_complete(dm.create())
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return future.result()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await asyncio.gather(*futures)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await self.run()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree): pk=1dddfb60-f80f-4842-a791-2
803f6e4751a>"
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: Traceback (most recent call last):
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: rv = job.perform()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: self._result = self._execute()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return self.func(*self.args, **self.kwargs)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: migrate_content(plan)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: plugin.migrator.migrate_content_to_pulp3()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: loop.run_until_complete(dm.create())
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: return future.result()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await asyncio.gather(*futures)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: await self.run()
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 293, in _needed_remote_artifacts
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Oct 09 02:18:16 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[6726]: ValueError: No declared artifact with relative path "images/boot.iso" for content "<Content (pulp_type=rpm.distribution_tree): pk=1dddfb60-f80f-4842-a791-2
803f6e4751a>"

Possibly related to https://github.com/pulp/pulp-2to3-migration/pull/240/ ?

Also possibly related to https://pulp.plan.io/issues/6944

If the repository is part of the migration plan, it successfully migrates. Also if both repositories are available during the first migration it successfully migrates.


Files

7681_no_declared_artifact.sh (1.52 KB) 7681_no_declared_artifact.sh dalley, 11/12/2020 04:04 AM
Actions #1

Updated by dalley over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #2

Updated by dalley over 3 years ago

  • Description updated (diff)
Actions #3

Updated by ttereshc over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 83
Actions #4

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 83 to Sprint 84
Actions #5

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 84 to Sprint 85
Actions #6

Updated by dalley over 3 years ago

Reproducer script attached

Actions #7

Updated by ttereshc over 3 years ago

  • Tags Katello added
Actions #8

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 85 to Sprint 86
Actions #9

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 86 to Sprint 87
Actions #10

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 87 to Sprint 88
Actions #11

Updated by ttereshc about 3 years ago

  • Sprint/Milestone set to 0.9.0
Actions #12

Updated by dalley about 3 years ago

It looks like the .treeinfo declarative artifact is attached to a declarative content where the contentartifact relative path is images/boot.iso

Actions #13

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 88 to Sprint 89
Actions #14

Updated by dalley about 3 years ago

        for d_content in batch:
            for content_artifact in d_content.content._remote_artifact_saver_cas:   #  <--- content_artifact.relative_path == 'images/boot.iso'
                for d_artifact in d_content.d_artifacts:  # d_content.d_artifacts == [ {{ca: relative_path=='.treeinfo'}} ]    # only one item, no match, "else" is triggered
                    if d_artifact.relative_path == content_artifact.relative_path:
                        break
                else:
                    msg = _('No declared artifact with relative path "{rp}" for content "{c}"')
                    raise ValueError(
                        msg.format(rp=content_artifact.relative_path, c=d_content.content)
                    )
                for remote_artifact in content_artifact._remote_artifact_saver_ras:
                    if remote_artifact.remote_id == d_artifact.remote.pk:
                        break
                else:
                    if d_artifact.remote:
                        remote_artifact = self._create_remote_artifact(d_artifact, content_artifact)
                        needed_ras.append(remote_artifact)
        return needed_ras
[c.relative_path for c in ContentArtifact.objects.filter(content=d_content.content)]
Out[9]: 
['images/boot.iso',
 'images/efiboot.img',
 'images/install.img',
 'images/pxeboot/initrd.img',
 'images/pxeboot/vmlinuz',
 '.treeinfo']
[c.relative_path for c in d_content.content._remote_artifact_saver_cas]
Out[10]: 
['images/boot.iso',
 'images/efiboot.img',
 'images/pxeboot/initrd.img',
 'images/pxeboot/vmlinuz',
 'images/install.img',
 '.treeinfo']
[d.relative_path for d in d_content.d_artifacts]
Out[11]: ['.treeinfo']

From this info, it looks like the problem is that there are several content artifacts that don't have matching items in d_artifacts

Actions #15

Updated by dalley about 3 years ago

Which happens because both "downloaded" and "remote" are False, so it skips over the artifacts with "cannot download" warning.

Which happens because the Pulp2Importer isn't available because it wasn't in the migration plan.

So the "real" problem is that it's not skipping the entire DistributionTree content unit when one of the artifacts can't be found, it's just skipping the artifacts.

Actions #16

Updated by dalley about 3 years ago

  • Status changed from ASSIGNED to POST

Added by dalley about 3 years ago

Revision 989cbe23 | View on GitHub

Skip multi-artifact content properly when an artifact is unavailable.

closes: #7681 https://pulp.plan.io/issues/7681

Added by dalley about 3 years ago

Revision 989cbe23 | View on GitHub

Skip multi-artifact content properly when an artifact is unavailable.

closes: #7681 https://pulp.plan.io/issues/7681

Added by dalley about 3 years ago

Revision 989cbe23 | View on GitHub

Skip multi-artifact content properly when an artifact is unavailable.

closes: #7681 https://pulp.plan.io/issues/7681

Actions #17

Updated by dalley about 3 years ago

  • Status changed from POST to MODIFIED
Actions #18

Updated by ttereshc about 3 years ago

  • Sprint/Milestone changed from 0.9.0 to 0.7.0
Actions #19

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF