Issue #7342: file:// repo syncing fails with unexpected keyword argument 'silence_errors_for_response_status_codes' - RPM Support - Pulp

Actions

Send by e-mail Copy link

Issue #7342

closed

file:// repo syncing fails with unexpected keyword argument 'silence_errors_for_response_status_codes'

Added by jsherril@redhat.com over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

dalley

Sprint/Milestone:

3.6.1

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Quarter:

Description

Try to sync a remote/repo located on local disk, like: file:///var/lib/pulp/sync_imports/test_repos/zoo/

and sync it, with 3.6.0, you will get:

        File "/usr/local/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
          rv = job.perform()
        File "/usr/local/lib/python3.6/site-packages/rq/job.py", line 657, in perform
          self._result = self._execute()
        File "/usr/local/lib/python3.6/site-packages/rq/job.py", line 663, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 220, in synchronize
          treeinfo = get_treeinfo_data(remote, remote_url)
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/kickstart/treeinfo.py", line 25, in get_treeinfo_data
          url=urljoin(remote_url, namespace), silence_errors_for_response_status_codes={404}
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 103, in get_downloader
          return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 306, in get_downloader
          return self.download_factory.build(url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 148, in build
          return builder(download_class, url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 191, in _generic
          return download_class(url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/file.py", line 40, in __init__
          super().__init__(url, **kwargs)
    description: __init__() got an unexpected keyword argument 'silence_errors_for_response_status_codes'

likely related to https://github.com/pulp/pulp_rpm/commit/91e8f7c65bd92f1587d67385ebf4d81bca50a219

Related issues

Actions

Copy link

Updated by jsherril@redhat.com over 3 years ago

Project changed from Pulp to RPM Support

Actions

Copy link

Updated by bmbouter over 3 years ago

I don't understand how this error is occurring because the base FileDownloader implemetnation already accepts **kwargs here.

Actions

Copy link

Updated by fao89 over 3 years ago

we need to add tests for syncing local repos, so we can catch these errors earlier

Actions

Copy link

Updated by dalley over 3 years ago

Status changed from NEW to ASSIGNED
Assignee set to dalley

Actions

Copy link

Updated by dalley over 3 years ago

Related to Test #7350: Test syncing from a repository located on a local disk added

Actions

Copy link

Updated by dalley over 3 years ago

@fabricio, I agree, I just filed a new test issue for this. https://pulp.plan.io/issues/7350

Actions

Copy link

Updated by bmbouter over 3 years ago

Oh I see why, even though FileDownloader accepts kwargs, the BaseDownloader does not https://github.com/pulp/pulpcore/blob/b94abd64d76ea4554e6750ff38ce458eaa888cc8/pulpcore/download/base.py#L66-L73. This is by design because if an additional kwarg makes it's way all the way to BaseDownloader then the factory has not done it's job fully.

Actions

Copy link

Updated by dalley over 3 years ago

bmbouter, yeah, but it should be filtered out before that point by the RpmDownloader initializer. Which makes me think that somehow RpmDownloader class isn't being used somehow.

I'm having difficulty creating the remote to reproduce this, is there something special I need to do?

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ sudo cp -R fixtures.pulpproject.org/ /var/lib/pulp/                                                                                                                                    
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ ls /var/lib/pulp
0005_puppet_module_name_change.txt  assets  content  db_initialized.flag  fixtures.pulpproject.org  published  static  tmp  uploads
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http POST :24817/pulp/api/v3/remotes/rpm/rpm/ name=localtest url="file:///var/lib/pulp/fixtures.pulpproject.org/rpm-unsigned/"                                                         
HTTP/1.1 400 Bad Request
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 107
Content-Type: application/json
Date: Wed, 19 Aug 2020 16:16:48 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "url": [
        "url 'file:///var/lib/pulp/fixtures.pulpproject.org/rpm-unsigned/' is not an allowed import path"
    ]
}

Actions

Copy link

Updated by bmbouter over 3 years ago

I recommend pulp_rpm to use a more customized DownloaderFactory to ensure the kwarg is only being passed to HttpDownloader. It would probably optionally strip the kwarg from FileDownloader by overridding DownloaderFactory._generic https://github.com/pulp/pulpcore/blob/387de62ea07bcdc16f19a4914368208a15182a49/pulpcore/download/factory.py#L176-L191

That subclass can be used by pulp_rpm here: https://github.com/pulp/pulp_rpm/blob/bb502a6efe3a83e2a58490f657b1024207731300/pulp_rpm/app/models/repository.py#L60-L79

Actions

Copy link

#10