Project

Profile

Help

Issue #7342

file:// repo syncing fails with unexpected keyword argument 'silence_errors_for_response_status_codes'

Added by jsherril@redhat.com over 1 year ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Try to sync a remote/repo located on local disk, like: file:///var/lib/pulp/sync_imports/test_repos/zoo/

and sync it, with 3.6.0, you will get:

        File "/usr/local/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
          rv = job.perform()
        File "/usr/local/lib/python3.6/site-packages/rq/job.py", line 657, in perform
          self._result = self._execute()
        File "/usr/local/lib/python3.6/site-packages/rq/job.py", line 663, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 220, in synchronize
          treeinfo = get_treeinfo_data(remote, remote_url)
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/kickstart/treeinfo.py", line 25, in get_treeinfo_data
          url=urljoin(remote_url, namespace), silence_errors_for_response_status_codes={404}
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 103, in get_downloader
          return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 306, in get_downloader
          return self.download_factory.build(url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 148, in build
          return builder(download_class, url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 191, in _generic
          return download_class(url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/file.py", line 40, in __init__
          super().__init__(url, **kwargs)
    description: __init__() got an unexpected keyword argument 'silence_errors_for_response_status_codes'

likely related to https://github.com/pulp/pulp_rpm/commit/91e8f7c65bd92f1587d67385ebf4d81bca50a219


Related issues

Related to RPM Support - Test #7350: Test syncing from a repository located on a local diskMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Refactor #7352: Provide a better abstraction for customizing downloader behaviorNEW

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 8ef4e58d View on GitHub
Added by dalley over 1 year ago

Fix sync from local (on-disk) repository

[nocoverage]

closes: #7342 https://pulp.plan.io/issues/7342

Revision a517e0b8 View on GitHub
Added by dalley over 1 year ago

Fix sync from local (on-disk) repository

[nocoverage]

closes: #7342 https://pulp.plan.io/issues/7342 (cherry picked from commit 8ef4e58d6aad34a8a0ca9ca6cca4311faef47041)

History

#1 Updated by jsherril@redhat.com over 1 year ago

  • Project changed from Pulp to RPM Support

#2 Updated by bmbouter over 1 year ago

I don't understand how this error is occurring because the base FileDownloader implemetnation already accepts **kwargs here.

#3 Updated by fao89 over 1 year ago

we need to add tests for syncing local repos, so we can catch these errors earlier

#4 Updated by dalley over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#5 Updated by dalley over 1 year ago

  • Related to Test #7350: Test syncing from a repository located on a local disk added

#6 Updated by dalley over 1 year ago

@fabricio, I agree, I just filed a new test issue for this. https://pulp.plan.io/issues/7350

#7 Updated by bmbouter over 1 year ago

Oh I see why, even though FileDownloader accepts kwargs, the BaseDownloader does not https://github.com/pulp/pulpcore/blob/b94abd64d76ea4554e6750ff38ce458eaa888cc8/pulpcore/download/base.py#L66-L73. This is by design because if an additional kwarg makes it's way all the way to BaseDownloader then the factory has not done it's job fully.

#8 Updated by dalley over 1 year ago

bmbouter, yeah, but it should be filtered out before that point by the RpmDownloader initializer. Which makes me think that somehow RpmDownloader class isn't being used somehow.

I'm having difficulty creating the remote to reproduce this, is there something special I need to do?

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ sudo cp -R fixtures.pulpproject.org/ /var/lib/pulp/                                                                                                                                    
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ ls /var/lib/pulp
0005_puppet_module_name_change.txt  assets  content  db_initialized.flag  fixtures.pulpproject.org  published  static  tmp  uploads
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http POST :24817/pulp/api/v3/remotes/rpm/rpm/ name=localtest url="file:///var/lib/pulp/fixtures.pulpproject.org/rpm-unsigned/"                                                         
HTTP/1.1 400 Bad Request
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 107
Content-Type: application/json
Date: Wed, 19 Aug 2020 16:16:48 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "url": [
        "url 'file:///var/lib/pulp/fixtures.pulpproject.org/rpm-unsigned/' is not an allowed import path"
    ]
}


#9 Updated by bmbouter over 1 year ago

I recommend pulp_rpm to use a more customized DownloaderFactory to ensure the kwarg is only being passed to HttpDownloader. It would probably optionally strip the kwarg from FileDownloader by overridding DownloaderFactory._generic https://github.com/pulp/pulpcore/blob/387de62ea07bcdc16f19a4914368208a15182a49/pulpcore/download/factory.py#L176-L191

That subclass can be used by pulp_rpm here: https://github.com/pulp/pulp_rpm/blob/bb502a6efe3a83e2a58490f657b1024207731300/pulp_rpm/app/models/repository.py#L60-L79

#10 Updated by bmbouter over 1 year ago

#11 Updated by pulpbot over 1 year ago

  • Status changed from ASSIGNED to POST

#12 Updated by dalley over 1 year ago

  • Related to Refactor #7352: Provide a better abstraction for customizing downloader behavior added

#13 Updated by dalley over 1 year ago

  • Status changed from POST to MODIFIED

#14 Updated by ttereshc over 1 year ago

  • Sprint/Milestone set to 3.6.1

#15 Updated by pulpbot over 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF