Project

Profile

Help

Issue #7208

closed

Error during sync : Path is duplicated

Added by ipanova@redhat.com over 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 101
Quarter:

Description

Sync repo http://ftp.cs.stanford.edu/centos/7/opstools/x86_64/ and observe traceback

$ http GET $BASE_ADDR/pulp/api/v3/tasks/91ae8e5d-bb4f-4151-a5c3-d24a3c04d81e/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2558
Content-Type: application/json
Date: Fri, 24 Jul 2020 10:42:54 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Cannot create repository version. Path is duplicated: rubygem-elasticsearch-doc-1.0.8-1.el7.noarch.rpm.",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 206, in synchronize\n    dv.create()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n    loop.run_until_complete(pipeline)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py\", line 775, in __exit__\n    repository.finalize_new_version(self)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py\", line 145, in finalize_new_version\n    validate_repo_version(new_version)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py\", line 141, in validate_repo_version\n    validate_version_paths(version)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py\", line 128, in validate_version_paths\n    raise ValueError(_(\"Cannot create repository version. {err}.\").format(err=e))\n"
    },
    "finished_at": "2020-07-24T10:40:27.431030Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "downloading.metadata",
            "done": 4,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "downloading.artifacts",
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 1069,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.packages",
            "done": 1069,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 1069
        }
    ],
    "pulp_created": "2020-07-24T10:40:14.425149Z",
    "pulp_href": "/pulp/api/v3/tasks/91ae8e5d-bb4f-4151-a5c3-d24a3c04d81e/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/57f994cf-4caf-4c37-b631-0eefdab2b8a1/",
        "/pulp/api/v3/repositories/rpm/rpm/9da9008b-937b-468a-bbaa-f3786b4b7b15/"
    ],
    "started_at": "2020-07-24T10:40:14.597862Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/9ab80819-e321-4187-83ef-4d8adafd3d80/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 _scripts]$ 

Related issues

Related to Pulp - Issue #8133: Same package from different repos yields error on syncCLOSED - CURRENTRELEASEggaineyActions
Has duplicate RPM Support - Issue #8043: ValueError: Path is duplicated when syncing http://mirror.centos.org/centos/7/opstools/x86_64/CLOSED - DUPLICATEActions
Has duplicate RPM Support - Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicatedCLOSED - DUPLICATEActions
Copied to RPM Support - Backport #9192: Backport #7208 "Error during sync : Path is duplicated" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Copied to RPM Support - Backport #9285: Backport 'Error during sync : Path is duplicated' to 3.11.zCLOSED - CURRENTRELEASEttereshc

Actions
Actions #1

Updated by ttereshc over 3 years ago

There are 2 of them indeed, different by location_href and checksum :/

<package type="rpm">
  <name>rubygem-elasticsearch-doc</name>
  <arch>noarch</arch>
  <version epoch="0" ver="1.0.8" rel="1.el7"/>
  <checksum type="sha256" pkgid="YES">beb1c816b3df53b53e544865eb4a823989522627ac905db6c5061b45bc2f4de0</checksum>
  <summary>Documentation for rubygem-elasticsearch</summary>
  <description>Documentation for rubygem-elasticsearch.</description>
  <packager>CBS &lt;cbs@centos.org&gt;</packager>
  <url>http://github.com/elasticsearch/elasticsearch-ruby</url>
  <time file="1493661231" build="1460639359"/>
  <size package="72024" installed="177000" archive="187056"/>
<location href="fluentd/rubygem-elasticsearch-doc-1.0.8-1.el7.noarch.rpm"/>
...
<package type="rpm">
  <name>rubygem-elasticsearch-doc</name>
  <arch>noarch</arch>
  <version epoch="0" ver="1.0.8" rel="1.el7"/>
  <checksum type="sha256" pkgid="YES">5be0c906ff3c21e2755c1b60da5011cb771147dfa27e817f12a7488803e71d0c</checksum>
  <summary>Documentation for rubygem-elasticsearch</summary>
  <description>Documentation for rubygem-elasticsearch.</description>
  <packager>CBS &lt;cbs@centos.org&gt;</packager>
  <url>http://github.com/elasticsearch/elasticsearch-ruby</url>
  <time file="1497443660" build="1460639359"/>
  <size package="72024" installed="177000" archive="187056"/>
<location href="logging/rubygem-elasticsearch-doc-1.0.8-1.el7.noarch.rpm"/>
Actions #2

Updated by ttereshc over 3 years ago

as for dnf behaviour, so far I got " ttereshc: When choices are identical, then solver could pick random option."

Actions #3

Updated by ttereshc over 3 years ago

  • Triaged changed from No to Yes
Actions #4

Updated by ggainey about 3 years ago

  • Has duplicate Issue #8043: ValueError: Path is duplicated when syncing http://mirror.centos.org/centos/7/opstools/x86_64/ added
Actions #5

Updated by dalley about 3 years ago

  • Related to Issue #8133: Same package from different repos yields error on sync added
Actions #6

Updated by dalley about 3 years ago

  • Related to Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicated added
Actions #7

Updated by adam.winberg@smhi.se about 3 years ago

Is there any workaround to this? We have repos we cant sync because of this. When trying to remove the repos and re-add them we run into #8351. So we are stuck between bugs and it's becoming a real problem for us.

Actions #8

Updated by wibbit over 2 years ago

Morning all.

I'm also seeing this issue, so would be keen to see a resolution.

I'm hitting it on

{'ca_cert': None, 'client_cert': None, 'connect_timeout': None, 'download_concurrency': 10, 'headers': None, 'name': 'upstream_rhel7_influxdb', 'policy': 'immediate', 'proxy_url': None, 'pulp_created': datetime.datetime(2021, 5, 10, 19, 4, 10, 745798, tzinfo=tzutc()), 'pulp_href': '/pulp/api/v3/remotes/rpm/rpm/47e8b4af-2ff4-4b89-945a-51b5201c424c/', 'pulp_labels': {}, 'pulp_last_updated': datetime.datetime(2021, 7, 9, 9, 8, 49, 768788, tzinfo=tzutc()), 'rate_limit': None, 'sles_auth_token': None, 'sock_connect_timeout': None, 'sock_read_timeout': None, 'tls_validation': True, 'total_timeout': None, 'url': 'https://repos.influxdata.com/rhel/7Server/x86_64/stable/'}

Any thing I can do to help let me know.

Actions #9

Updated by pulpbot over 2 years ago

  • Status changed from NEW to POST
Actions #10

Updated by ttereshc over 2 years ago

  • Assignee set to ggainey
  • Sprint set to Sprint 101
Actions #11

Updated by ttereshc over 2 years ago

  • Related to deleted (Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicated)
Actions #12

Updated by ttereshc over 2 years ago

  • Has duplicate Issue #7507: Pulp RPM 3.6.2 - ValueError: Cannot create repository version. Path is duplicated added

Added by ggainey over 2 years ago

Revision 0152ff42 | View on GitHub

Make "is this version valid" checks more lenient.

Handles the case of "same name, multiple locations, diff content".

fixes #7208

Actions #14

Updated by ggainey over 2 years ago

  • Status changed from POST to MODIFIED
Actions #15

Updated by dalley over 2 years ago

  • Sprint/Milestone set to 3.15.0
Actions #16

Updated by dalley over 2 years ago

  • Copied to Backport #9192: Backport #7208 "Error during sync : Path is duplicated" to 3.14.z added
Actions #17

Updated by ttereshc over 2 years ago

  • Copied to Backport #9285: Backport 'Error during sync : Path is duplicated' to 3.11.z added
Actions #18

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF