Issue #9003
closedRepo sync fails with '__init__() got an unexpected keyword argument 'max_retries''
Description
Syncing this repo:
{ "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/", "pulp_created": "2021-07-02T14:01:38.561803Z", "versions_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/versions/", "pulp_labels": {}, "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/versions/0/", "name": "2_duplicate", "description": null, "retained_versions": null, "remote": null, "autopublish": false, "metadata_signing_service": null, "retain_package_versions": 0, "metadata_checksum_type": null, "package_checksum_type": null, "gpgcheck": 0, "repo_gpgcheck": 0, "sqlite_metadata": false },
with this remote:
{ "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/50a42d67-c219-4303-85ae-5f2d5c33c384/", "pulp_created": "2021-07-02T14:01:38.713952Z", "name": "2_duplicate", "url": "file:///var/lib/pulp/sync_imports/test_repos/zoo/", "ca_cert": null, "client_cert": null, "tls_validation": true, "proxy_url": null, "pulp_labels": {}, "pulp_last_updated": "2021-07-02T14:01:38.713968Z", "download_concurrency": null, "max_retries": null, "policy": "immediate", "total_timeout": 300.0, "connect_timeout": null, "sock_connect_timeout": null, "sock_read_timeout": null, "headers": null, "rate_limit": null, "sles_auth_token": null }
fails with the following backtrace:
Jul 2 14:01:40 localhost pulpcore-worker-2: pulp [3f8b28ace1024b8bb2d379af1178341f]: pulpcore.tasking.pulpcore_worker:INFO: Task 07bb5995-77a4-4cf3-ae87-eaa66c7b8597 failed (__init__() got an unexpected keyword argument 'max_retries') Jul 2 14:01:40 localhost pulpcore-worker-2: pulp [3f8b28ace1024b8bb2d379af1178341f]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 266, in _perform_task Jul 2 14:01:40 localhost pulpcore-worker-2: result = func(*args, **kwargs) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 338, in synchronize Jul 2 14:01:40 localhost pulpcore-worker-2: remote_url = fetch_remote_url(remote) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 230, in fetch_remote_url Jul 2 14:01:40 localhost pulpcore-worker-2: if get_repomd_file(remote, remote_url): Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 189, in get_repomd_file Jul 2 14:01:40 localhost pulpcore-worker-2: downloader = remote.get_downloader(url=urlpath_sanitize(url, "repodata/repomd.xml")) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 100, in get_downloader Jul 2 14:01:40 localhost pulpcore-worker-2: return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 410, in get_downloader Jul 2 14:01:40 localhost pulpcore-worker-2: return self.download_factory.build(url, **kwargs) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 176, in build Jul 2 14:01:40 localhost pulpcore-worker-2: return builder(download_class, url, **kwargs) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 225, in _generic Jul 2 14:01:40 localhost pulpcore-worker-2: return download_class(url, **kwargs) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 32, in __init__ Jul 2 14:01:40 localhost pulpcore-worker-2: super().__init__(*args, **kwargs) Jul 2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/file.py", line 40, in __init__ Jul 2 14:01:40 localhost pulpcore-worker-2: super().__init__(url, **kwargs)
Not all RPM repo syncs fail. This one in particular syncs from the filesystem. Perhaps that is related?
Pulp 3 versions:
pulp-2to3-migration (0.12.0) pulp-ansible (0.8.0) pulp-certguard (1.4.0) pulp-cli (0.10.1) pulp-container (2.7.0) pulp-deb (2.13.0) pulp-file (1.8.1) pulp-python (3.4.0) pulp-rpm (3.13.2) pulpcore (3.14.0)
Related issues
Updated by dalley over 3 years ago
- Project changed from RPM Support to Pulp
- Assignee set to dalley
Updated by dalley over 3 years ago
- Copied to Backport #9015: Backport fix for 9003 to 3.14.z - unable to sync from repos on the filesystem added
Updated by dalley over 3 years ago
- Related to Refactor #7352: Provide a better abstraction for customizing downloader behavior added
Updated by dalley over 3 years ago
This line [0] injects the max_retries
keyword argument via the factory, but only HttpDownloader
subclasses are capable of handling it. Direct BaseDownloader
subclasses such as FileDownloader
get it but don't have any way to handle it.
We can't move it outside of the downloader factory build() function because that is the last place in the chain with access to the remote, which is where this data is stored. This comes back to our abstractions in this area not being very cleanly separated (attached issue).
So until we can refactor it, the fix is to declare *args, **kwargs
on BaseDownloader
so that any extra kwargs are simply ignored.
Updated by pulpbot over 3 years ago
Added by dalley over 3 years ago
Updated by dalley over 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulpcore|e4b17247d30b85841e5065d745a1fbb219a58cec.
Updated by pulpbot over 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Fix syncs of repos from file:// urls
closes: #9003 https://pulp.plan.io/issues/9003