Project

Profile

Help

Issue #9003

closed

Repo sync fails with '__init__() got an unexpected keyword argument 'max_retries''

Added by iballou almost 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Syncing this repo:

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/",
    "pulp_created": "2021-07-02T14:01:38.561803Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/versions/0/",
    "name": "2_duplicate",
    "description": null,
    "retained_versions": null,
    "remote": null,
    "autopublish": false,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": null,
    "package_checksum_type": null,
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

with this remote:

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/50a42d67-c219-4303-85ae-5f2d5c33c384/",
    "pulp_created": "2021-07-02T14:01:38.713952Z",
    "name": "2_duplicate",
    "url": "file:///var/lib/pulp/sync_imports/test_repos/zoo/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-07-02T14:01:38.713968Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "immediate",
    "total_timeout": 300.0,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": null,
    "rate_limit": null,
    "sles_auth_token": null
  }

fails with the following backtrace:



        Jul  2 14:01:40 localhost pulpcore-worker-2: pulp [3f8b28ace1024b8bb2d379af1178341f]: pulpcore.tasking.pulpcore_worker:INFO: Task 07bb5995-77a4-4cf3-ae87-eaa66c7b8597 failed (__init__() got an unexpected keyword argument 'max_retries')
        Jul  2 14:01:40 localhost pulpcore-worker-2: pulp [3f8b28ace1024b8bb2d379af1178341f]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 266, in _perform_task
        Jul  2 14:01:40 localhost pulpcore-worker-2: result = func(*args, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 338, in synchronize
        Jul  2 14:01:40 localhost pulpcore-worker-2: remote_url = fetch_remote_url(remote)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 230, in fetch_remote_url
        Jul  2 14:01:40 localhost pulpcore-worker-2: if get_repomd_file(remote, remote_url):
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 189, in get_repomd_file
        Jul  2 14:01:40 localhost pulpcore-worker-2: downloader = remote.get_downloader(url=urlpath_sanitize(url, "repodata/repomd.xml"))
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 100, in get_downloader
        Jul  2 14:01:40 localhost pulpcore-worker-2: return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 410, in get_downloader
        Jul  2 14:01:40 localhost pulpcore-worker-2: return self.download_factory.build(url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 176, in build
        Jul  2 14:01:40 localhost pulpcore-worker-2: return builder(download_class, url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 225, in _generic
        Jul  2 14:01:40 localhost pulpcore-worker-2: return download_class(url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 32, in __init__
        Jul  2 14:01:40 localhost pulpcore-worker-2: super().__init__(*args, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/file.py", line 40, in __init__
        Jul  2 14:01:40 localhost pulpcore-worker-2: super().__init__(url, **kwargs)

Not all RPM repo syncs fail. This one in particular syncs from the filesystem. Perhaps that is related?

Pulp 3 versions:

pulp-2to3-migration (0.12.0)
pulp-ansible (0.8.0)
pulp-certguard (1.4.0)
pulp-cli (0.10.1)
pulp-container (2.7.0)
pulp-deb (2.13.0)
pulp-file (1.8.1)
pulp-python (3.4.0)
pulp-rpm (3.13.2)
pulpcore (3.14.0)

Related issues

Related to Pulp - Refactor #7352: Provide a better abstraction for customizing downloader behaviorCLOSED - DUPLICATE

Actions
Copied to Pulp - Backport #9015: Backport fix for 9003 to 3.14.z - unable to sync from repos on the filesystemCLOSED - CURRENTRELEASE

Actions
Actions #1

Updated by dalley almost 3 years ago

  • Project changed from RPM Support to Pulp
  • Assignee set to dalley
Actions #2

Updated by dalley almost 3 years ago

  • Status changed from NEW to POST
Actions #3

Updated by dalley almost 3 years ago

  • Copied to Backport #9015: Backport fix for 9003 to 3.14.z - unable to sync from repos on the filesystem added
Actions #4

Updated by dalley almost 3 years ago

  • Related to Refactor #7352: Provide a better abstraction for customizing downloader behavior added
Actions #5

Updated by dalley almost 3 years ago

This line [0] injects the max_retries keyword argument via the factory, but only HttpDownloader subclasses are capable of handling it. Direct BaseDownloader subclasses such as FileDownloader get it but don't have any way to handle it.

We can't move it outside of the downloader factory build() function because that is the last place in the chain with access to the remote, which is where this data is stored. This comes back to our abstractions in this area not being very cleanly separated (attached issue).

So until we can refactor it, the fix is to declare *args, **kwargs on BaseDownloader so that any extra kwargs are simply ignored.

https://github.com/pulp/pulpcore/pull/1401/files#diff-e73310cead1782834422ae6ace57aada58f37ba99bbb01c443a6bd1cbc60e26dR163

Added by dalley almost 3 years ago

Revision e4b17247 | View on GitHub

Fix syncs of repos from file:// urls

closes: #9003 https://pulp.plan.io/issues/9003

Actions #7

Updated by dalley almost 3 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by pulpbot over 2 years ago

  • Sprint/Milestone set to 3.15.0
Actions #9

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF