Project

Profile

Help

Issue #9003

Repo sync fails with '__init__() got an unexpected keyword argument 'max_retries''

Added by iballou 3 months ago. Updated 21 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Syncing this repo:

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/",
    "pulp_created": "2021-07-02T14:01:38.561803Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/fb30e326-0897-416d-9862-a31193e12317/versions/0/",
    "name": "2_duplicate",
    "description": null,
    "retained_versions": null,
    "remote": null,
    "autopublish": false,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": null,
    "package_checksum_type": null,
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

with this remote:

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/50a42d67-c219-4303-85ae-5f2d5c33c384/",
    "pulp_created": "2021-07-02T14:01:38.713952Z",
    "name": "2_duplicate",
    "url": "file:///var/lib/pulp/sync_imports/test_repos/zoo/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-07-02T14:01:38.713968Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "immediate",
    "total_timeout": 300.0,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": null,
    "rate_limit": null,
    "sles_auth_token": null
  }

fails with the following backtrace:



        Jul  2 14:01:40 localhost pulpcore-worker-2: pulp [3f8b28ace1024b8bb2d379af1178341f]: pulpcore.tasking.pulpcore_worker:INFO: Task 07bb5995-77a4-4cf3-ae87-eaa66c7b8597 failed (__init__() got an unexpected keyword argument 'max_retries')
        Jul  2 14:01:40 localhost pulpcore-worker-2: pulp [3f8b28ace1024b8bb2d379af1178341f]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 266, in _perform_task
        Jul  2 14:01:40 localhost pulpcore-worker-2: result = func(*args, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 338, in synchronize
        Jul  2 14:01:40 localhost pulpcore-worker-2: remote_url = fetch_remote_url(remote)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 230, in fetch_remote_url
        Jul  2 14:01:40 localhost pulpcore-worker-2: if get_repomd_file(remote, remote_url):
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 189, in get_repomd_file
        Jul  2 14:01:40 localhost pulpcore-worker-2: downloader = remote.get_downloader(url=urlpath_sanitize(url, "repodata/repomd.xml"))
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 100, in get_downloader
        Jul  2 14:01:40 localhost pulpcore-worker-2: return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 410, in get_downloader
        Jul  2 14:01:40 localhost pulpcore-worker-2: return self.download_factory.build(url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 176, in build
        Jul  2 14:01:40 localhost pulpcore-worker-2: return builder(download_class, url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 225, in _generic
        Jul  2 14:01:40 localhost pulpcore-worker-2: return download_class(url, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 32, in __init__
        Jul  2 14:01:40 localhost pulpcore-worker-2: super().__init__(*args, **kwargs)
        Jul  2 14:01:40 localhost pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulpcore/download/file.py", line 40, in __init__
        Jul  2 14:01:40 localhost pulpcore-worker-2: super().__init__(url, **kwargs)

Not all RPM repo syncs fail. This one in particular syncs from the filesystem. Perhaps that is related?

Pulp 3 versions:

pulp-2to3-migration (0.12.0)
pulp-ansible (0.8.0)
pulp-certguard (1.4.0)
pulp-cli (0.10.1)
pulp-container (2.7.0)
pulp-deb (2.13.0)
pulp-file (1.8.1)
pulp-python (3.4.0)
pulp-rpm (3.13.2)
pulpcore (3.14.0)

Related issues

Related to Pulp - Refactor #7352: Provide a better abstraction for customizing downloader behaviorNEW

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to Pulp - Backport #9015: Backport fix for 9003 to 3.14.z - unable to sync from repos on the filesystemCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision e4b17247 View on GitHub
Added by dalley 2 months ago

Fix syncs of repos from file:// urls

closes: #9003 https://pulp.plan.io/issues/9003

History

#1 Updated by dalley 2 months ago

  • Project changed from RPM Support to Pulp
  • Assignee set to dalley

#2 Updated by dalley 2 months ago

  • Status changed from NEW to POST

#3 Updated by dalley 2 months ago

  • Copied to Backport #9015: Backport fix for 9003 to 3.14.z - unable to sync from repos on the filesystem added

#4 Updated by dalley 2 months ago

  • Related to Refactor #7352: Provide a better abstraction for customizing downloader behavior added

#5 Updated by dalley 2 months ago

This line [0] injects the max_retries keyword argument via the factory, but only HttpDownloader subclasses are capable of handling it. Direct BaseDownloader subclasses such as FileDownloader get it but don't have any way to handle it.

We can't move it outside of the downloader factory build() function because that is the last place in the chain with access to the remote, which is where this data is stored. This comes back to our abstractions in this area not being very cleanly separated (attached issue).

So until we can refactor it, the fix is to declare *args, **kwargs on BaseDownloader so that any extra kwargs are simply ignored.

https://github.com/pulp/pulpcore/pull/1401/files#diff-e73310cead1782834422ae6ace57aada58f37ba99bbb01c443a6bd1cbc60e26dR163

#7 Updated by dalley 2 months ago

  • Status changed from POST to MODIFIED

#8 Updated by pulpbot 21 days ago

  • Sprint/Milestone set to 3.15.0

#9 Updated by pulpbot 21 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF