Project

Profile

Help

Issue #8582

closed

ISO content migration fails with ValueError: Cannot create repository version. Path is duplicated: <filename>

Added by ttereshc almost 2 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 96
Quarter:

Description

In Pulp 2, it was possible to add N different files with the same filename (relative path) to the same repo.
Only one is published but according to the database there are N files with the same name in one repo.
For that reason, migration plugin tries to create a repo version with all N files with the same name. It can happen only on the system which was upgraded from pulp <= 2.13.3. Here is the failure:

 pulp: rq.worker:ERROR: Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/plugin/repo_version_utils.py", line 124, in validate_version_paths
 validate_file_paths(paths)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/files.py", line 133, in validate_file_paths
 raise ValueError(_("Path is duplicated: {path}").format(path=path))
 ValueError: Path is duplicated: katello-installer-3.0.0.88.tar.gz
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
 rv = job.perform()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
 self._result = self._execute()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
 return self.func(*self.args, **self.kwargs)
 File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 194, in complex_repo_migration
 create_repo_version(progress_rv, pulp2_repo, pulp3_remote)
 File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 344, in create_repo_version
 new_version.remove_content(Content.objects.filter(pk__in=to_delete))
 File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 795, in __exit__
 repository.finalize_new_version(self)
 File "/usr/lib/python3.6/site-packages/pulp_file/app/models.py", line 79, in finalize_new_version
 validate_repo_version(new_version)
 File "/usr/lib/python3.6/site-packages/pulpcore/plugin/repo_version_utils.py", line 139, in validate_repo_version
 validate_version_paths(version)
 File "/usr/lib/python3.6/site-packages/pulpcore/plugin/repo_version_utils.py", line 126, in validate_version_paths
 raise ValueError(_("Cannot create repository version. {err}.").format(err=e))
 ValueError: Cannot create repository version. Path is duplicated: katello-installer-3.0.0.88.tar.gz.

It's a Pulp 2 bug #2773 which was fixed in 2.13.3 but we should handle this case to improve customer experience. There is no easy workaround, no easy way for a user to figure out which file is published and which needs to be removed from a Pulp 2 repo to fix the problem.

Proposed solution

Check pulp 2 logic and make choice in the same way if it's deterministic. That will give a user exactly the same result as they have in Pulp 2.
If it's somewhat random in Pulp 2, choose the latest to add to a repo version.

Observation

It seems like Pulp 2 picks the first one(the oldest one) to publish, I haven't checked the code but the system I see the error on has the older file published.

Actions #1

Updated by ttereshc almost 2 years ago

  • Subject changed from migration fails with ValueError: Cannot create repository version. Path is duplicated: <filename> to ISO content migration fails with ValueError: Cannot create repository version. Path is duplicated: <filename>
Actions #2

Updated by ttereshc almost 2 years ago

  • Tags Katello added
Actions #3

Updated by ttereshc almost 2 years ago

  • Description updated (diff)
Actions #4

Updated by ttereshc almost 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
Actions #5

Updated by ttereshc almost 2 years ago

  • Sprint set to Sprint 95
Actions #6

Updated by rchan almost 2 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #7

Updated by ttereshc over 1 year ago

  • Status changed from ASSIGNED to POST

Added by ttereshc over 1 year ago

Revision 707518a1

Handle overlaping paths which come from pulp2 repos

closes #8582 https://pulp.plan.io/issues/8582

Added by ttereshc over 1 year ago

Revision 707518a1

Handle overlaping paths which come from pulp2 repos

closes #8582 https://pulp.plan.io/issues/8582

Added by ttereshc over 1 year ago

Revision 707518a1

Handle overlaping paths which come from pulp2 repos

closes #8582 https://pulp.plan.io/issues/8582

Actions #8

Updated by ttereshc over 1 year ago

  • Status changed from POST to MODIFIED
Actions #9

Updated by ttereshc over 1 year ago

  • Sprint/Milestone set to 0.11.1

Added by ttereshc over 1 year ago

Revision 4c0bee14

Handle overlaping paths which come from pulp2 repos

closes #8582 https://pulp.plan.io/issues/8582

(cherry picked from commit 707518a1dac56ba4b5748ed69b934185c52e84a1)

Added by ttereshc over 1 year ago

Revision 4c0bee14

Handle overlaping paths which come from pulp2 repos

closes #8582 https://pulp.plan.io/issues/8582

(cherry picked from commit 707518a1dac56ba4b5748ed69b934185c52e84a1)

Added by ttereshc over 1 year ago

Revision 4c0bee14

Handle overlaping paths which come from pulp2 repos

closes #8582 https://pulp.plan.io/issues/8582

(cherry picked from commit 707518a1dac56ba4b5748ed69b934185c52e84a1)

Actions #10

Updated by pulpbot over 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF