Project

Profile

Help

Maintenance: Planio will be observing a scheduled maintenance window this Sunday, November 10, 2024 from 20:00 UTC until 21:00 UTC to perform important network maintenance in our primary data center. Your Planio account will be unavailable for a few minutes during this maintenance window.

Issue #7342

closed

file:// repo syncing fails with unexpected keyword argument 'silence_errors_for_response_status_codes'

Added by jsherril@redhat.com about 4 years ago. Updated about 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Try to sync a remote/repo located on local disk, like: file:///var/lib/pulp/sync_imports/test_repos/zoo/

and sync it, with 3.6.0, you will get:

        File "/usr/local/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
          rv = job.perform()
        File "/usr/local/lib/python3.6/site-packages/rq/job.py", line 657, in perform
          self._result = self._execute()
        File "/usr/local/lib/python3.6/site-packages/rq/job.py", line 663, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 220, in synchronize
          treeinfo = get_treeinfo_data(remote, remote_url)
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/kickstart/treeinfo.py", line 25, in get_treeinfo_data
          url=urljoin(remote_url, namespace), silence_errors_for_response_status_codes={404}
        File "/usr/local/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 103, in get_downloader
          return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 306, in get_downloader
          return self.download_factory.build(url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 148, in build
          return builder(download_class, url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/factory.py", line 191, in _generic
          return download_class(url, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/download/file.py", line 40, in __init__
          super().__init__(url, **kwargs)
    description: __init__() got an unexpected keyword argument 'silence_errors_for_response_status_codes'

likely related to https://github.com/pulp/pulp_rpm/commit/91e8f7c65bd92f1587d67385ebf4d81bca50a219


Related issues

Related to RPM Support - Test #7350: Test syncing from a repository located on a local diskMODIFIEDdalleyActions
Related to Pulp - Refactor #7352: Provide a better abstraction for customizing downloader behaviorCLOSED - DUPLICATE

Actions
Actions #1

Updated by jsherril@redhat.com about 4 years ago

  • Project changed from Pulp to RPM Support
Actions #2

Updated by bmbouter about 4 years ago

I don't understand how this error is occurring because the base FileDownloader implemetnation already accepts **kwargs here.

Actions #3

Updated by fao89 about 4 years ago

we need to add tests for syncing local repos, so we can catch these errors earlier

Actions #4

Updated by dalley about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #5

Updated by dalley about 4 years ago

  • Related to Test #7350: Test syncing from a repository located on a local disk added
Actions #6

Updated by dalley about 4 years ago

@fabricio, I agree, I just filed a new test issue for this. https://pulp.plan.io/issues/7350

Actions #7

Updated by bmbouter about 4 years ago

Oh I see why, even though FileDownloader accepts kwargs, the BaseDownloader does not https://github.com/pulp/pulpcore/blob/b94abd64d76ea4554e6750ff38ce458eaa888cc8/pulpcore/download/base.py#L66-L73. This is by design because if an additional kwarg makes it's way all the way to BaseDownloader then the factory has not done it's job fully.

Actions #8

Updated by dalley about 4 years ago

bmbouter, yeah, but it should be filtered out before that point by the RpmDownloader initializer. Which makes me think that somehow RpmDownloader class isn't being used somehow.

I'm having difficulty creating the remote to reproduce this, is there something special I need to do?

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ sudo cp -R fixtures.pulpproject.org/ /var/lib/pulp/                                                                                                                                    
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ ls /var/lib/pulp
0005_puppet_module_name_change.txt  assets  content  db_initialized.flag  fixtures.pulpproject.org  published  static  tmp  uploads
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http POST :24817/pulp/api/v3/remotes/rpm/rpm/ name=localtest url="file:///var/lib/pulp/fixtures.pulpproject.org/rpm-unsigned/"                                                         
HTTP/1.1 400 Bad Request
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 107
Content-Type: application/json
Date: Wed, 19 Aug 2020 16:16:48 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "url": [
        "url 'file:///var/lib/pulp/fixtures.pulpproject.org/rpm-unsigned/' is not an allowed import path"
    ]
}


Actions #9

Updated by bmbouter about 4 years ago

I recommend pulp_rpm to use a more customized DownloaderFactory to ensure the kwarg is only being passed to HttpDownloader. It would probably optionally strip the kwarg from FileDownloader by overridding DownloaderFactory._generic https://github.com/pulp/pulpcore/blob/387de62ea07bcdc16f19a4914368208a15182a49/pulpcore/download/factory.py#L176-L191

That subclass can be used by pulp_rpm here: https://github.com/pulp/pulp_rpm/blob/bb502a6efe3a83e2a58490f657b1024207731300/pulp_rpm/app/models/repository.py#L60-L79

Actions #10

Updated by bmbouter about 4 years ago

Actions #11

Updated by pulpbot about 4 years ago

  • Status changed from ASSIGNED to POST
Actions #12

Updated by dalley about 4 years ago

  • Related to Refactor #7352: Provide a better abstraction for customizing downloader behavior added

Added by dalley about 4 years ago

Revision 8ef4e58d | View on GitHub

Fix sync from local (on-disk) repository

[nocoverage]

closes: #7342 https://pulp.plan.io/issues/7342

Actions #13

Updated by dalley about 4 years ago

  • Status changed from POST to MODIFIED

Added by dalley about 4 years ago

Revision a517e0b8 | View on GitHub

Fix sync from local (on-disk) repository

[nocoverage]

closes: #7342 https://pulp.plan.io/issues/7342 (cherry picked from commit 8ef4e58d6aad34a8a0ca9ca6cca4311faef47041)

Actions #14

Updated by ttereshc about 4 years ago

  • Sprint/Milestone set to 3.6.1
Actions #15

Updated by pulpbot about 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF