Project

Profile

Help

Issue #7208

closed

Error during sync : Path is duplicated

Added by ipanova@redhat.com almost 2 years ago. Updated 10 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 101
Quarter:

Description

Sync repo http://ftp.cs.stanford.edu/centos/7/opstools/x86_64/ and observe traceback

$ http GET $BASE_ADDR/pulp/api/v3/tasks/91ae8e5d-bb4f-4151-a5c3-d24a3c04d81e/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2558
Content-Type: application/json
Date: Fri, 24 Jul 2020 10:42:54 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Cannot create repository version. Path is duplicated: rubygem-elasticsearch-doc-1.0.8-1.el7.noarch.rpm.",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 206, in synchronize\n    dv.create()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n    loop.run_until_complete(pipeline)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py\", line 775, in __exit__\n    repository.finalize_new_version(self)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py\", line 145, in finalize_new_version\n    validate_repo_version(new_version)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py\", line 141, in validate_repo_version\n    validate_version_paths(version)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py\", line 128, in validate_version_paths\n    raise ValueError(_(\"Cannot create repository version. {err}.\").format(err=e))\n"
    },
    "finished_at": "2020-07-24T10:40:27.431030Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "downloading.metadata",
            "done": 4,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "downloading.artifacts",
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 1069,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.packages",
            "done": 1069,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 1069
        }
    ],
    "pulp_created": "2020-07-24T10:40:14.425149Z",
    "pulp_href": "/pulp/api/v3/tasks/91ae8e5d-bb4f-4151-a5c3-d24a3c04d81e/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/57f994cf-4caf-4c37-b631-0eefdab2b8a1/",
        "/pulp/api/v3/repositories/rpm/rpm/9da9008b-937b-468a-bbaa-f3786b4b7b15/"
    ],
    "started_at": "2020-07-24T10:40:14.597862Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/9ab80819-e321-4187-83ef-4d8adafd3d80/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 _scripts]$ 

Related issues

Related to Pulp - Issue #8133: Same package from different repos yields error on syncCLOSED - CURRENTRELEASEggaineyActions
Has duplicate RPM Support - Issue #8043: ValueError: Path is duplicated when syncing http://mirror.centos.org/centos/7/opstools/x86_64/CLOSED - DUPLICATEActions
Has duplicate RPM Support - Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicatedCLOSED - DUPLICATEActions
Copied to RPM Support - Backport #9192: Backport #7208 "Error during sync : Path is duplicated" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Copied to RPM Support - Backport #9285: Backport 'Error during sync : Path is duplicated' to 3.11.zCLOSED - CURRENTRELEASEttereshc

Actions
Actions #1

Updated by ttereshc almost 2 years ago

There are 2 of them indeed, different by location_href and checksum :/

<package type="rpm">
  <name>rubygem-elasticsearch-doc</name>
  <arch>noarch</arch>
  <version epoch="0" ver="1.0.8" rel="1.el7"/>
  <checksum type="sha256" pkgid="YES">beb1c816b3df53b53e544865eb4a823989522627ac905db6c5061b45bc2f4de0</checksum>
  <summary>Documentation for rubygem-elasticsearch</summary>
  <description>Documentation for rubygem-elasticsearch.</description>
  <packager>CBS &lt;cbs@centos.org&gt;</packager>
  <url>http://github.com/elasticsearch/elasticsearch-ruby</url>
  <time file="1493661231" build="1460639359"/>
  <size package="72024" installed="177000" archive="187056"/>
<location href="fluentd/rubygem-elasticsearch-doc-1.0.8-1.el7.noarch.rpm"/>
...
<package type="rpm">
  <name>rubygem-elasticsearch-doc</name>
  <arch>noarch</arch>
  <version epoch="0" ver="1.0.8" rel="1.el7"/>
  <checksum type="sha256" pkgid="YES">5be0c906ff3c21e2755c1b60da5011cb771147dfa27e817f12a7488803e71d0c</checksum>
  <summary>Documentation for rubygem-elasticsearch</summary>
  <description>Documentation for rubygem-elasticsearch.</description>
  <packager>CBS &lt;cbs@centos.org&gt;</packager>
  <url>http://github.com/elasticsearch/elasticsearch-ruby</url>
  <time file="1497443660" build="1460639359"/>
  <size package="72024" installed="177000" archive="187056"/>
<location href="logging/rubygem-elasticsearch-doc-1.0.8-1.el7.noarch.rpm"/>
Actions #2

Updated by ttereshc almost 2 years ago

as for dnf behaviour, so far I got " ttereshc: When choices are identical, then solver could pick random option."

Actions #3

Updated by ttereshc almost 2 years ago

  • Triaged changed from No to Yes
Actions #4

Updated by ggainey over 1 year ago

  • Has duplicate Issue #8043: ValueError: Path is duplicated when syncing http://mirror.centos.org/centos/7/opstools/x86_64/ added
Actions #5

Updated by dalley over 1 year ago

  • Related to Issue #8133: Same package from different repos yields error on sync added
Actions #6

Updated by dalley over 1 year ago

  • Related to Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicated added
Actions #7

Updated by adam.winberg@smhi.se over 1 year ago

Is there any workaround to this? We have repos we cant sync because of this. When trying to remove the repos and re-add them we run into #8351. So we are stuck between bugs and it's becoming a real problem for us.

Actions #8

Updated by wibbit 12 months ago

Morning all.

I'm also seeing this issue, so would be keen to see a resolution.

I'm hitting it on

{'ca_cert': None, 'client_cert': None, 'connect_timeout': None, 'download_concurrency': 10, 'headers': None, 'name': 'upstream_rhel7_influxdb', 'policy': 'immediate', 'proxy_url': None, 'pulp_created': datetime.datetime(2021, 5, 10, 19, 4, 10, 745798, tzinfo=tzutc()), 'pulp_href': '/pulp/api/v3/remotes/rpm/rpm/47e8b4af-2ff4-4b89-945a-51b5201c424c/', 'pulp_labels': {}, 'pulp_last_updated': datetime.datetime(2021, 7, 9, 9, 8, 49, 768788, tzinfo=tzutc()), 'rate_limit': None, 'sles_auth_token': None, 'sock_connect_timeout': None, 'sock_read_timeout': None, 'tls_validation': True, 'total_timeout': None, 'url': 'https://repos.influxdata.com/rhel/7Server/x86_64/stable/'}

Any thing I can do to help let me know.

Actions #9

Updated by pulpbot 12 months ago

  • Status changed from NEW to POST
Actions #10

Updated by ttereshc 11 months ago

  • Assignee set to ggainey
  • Sprint set to Sprint 101
Actions #11

Updated by ttereshc 11 months ago

  • Related to deleted (Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicated)
Actions #12

Updated by ttereshc 11 months ago

  • Has duplicate Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicated added

Added by ggainey 11 months ago

Revision 0152ff42

Make "is this version valid" checks more lenient.

Handles the case of "same name, multiple locations, diff content".

fixes #7208

Actions #14

Updated by ggainey 11 months ago

  • Status changed from POST to MODIFIED
Actions #15

Updated by dalley 11 months ago

  • Sprint/Milestone set to 3.15.0
Actions #16

Updated by dalley 11 months ago

  • Copied to Backport #9192: Backport #7208 "Error during sync : Path is duplicated" to 3.14.z added
Actions #17

Updated by ttereshc 11 months ago

  • Copied to Backport #9285: Backport 'Error during sync : Path is duplicated' to 3.11.z added
Actions #18

Updated by pulpbot 10 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF