Issue #8689: adding custom connection headers to debian remote causes repository syncs to always fail - Pulp

Actions

Send by e-mail Copy link

Issue #8689

closed

adding custom connection headers to debian remote causes repository syncs to always fail

Added by simon_t over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

dkliban@redhat.com

Category:

Sprint/Milestone:

3.13.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Sprint 97

Quarter:

Description

pulpcore version: 3.12.1 pulp_deb version: 2.11.1

In an attempt to resolve ongoing sync failures custom headers were added to a debian remote:

http patch https://pulphost.domain.local/pulp/api/v3/remotes/deb/apt/18600e5b-4914-4bbd-ae2a-bc0488c30e07/ headers:='[{"Connection": "keep-alive"}]'

Resulting in the following configuration which matches the format described in the pulp_deb API documentation:

{
    "architectures": "amd64",
    "ca_cert": null,
    "client_cert": null,
    "components": "main contrib non-free",
    "connect_timeout": null,
    "distributions": "bullseye",
    "download_concurrency": 10,
    "gpgkey": null,
    "headers": [
        {
            "Connection": "keep-alive"
        }
    ],
    "ignore_missing_package_indices": true,
    "name": "remote_debian_bullseye",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2021-04-29T11:17:38.824675Z",
    "pulp_href": "/pulp/api/v3/remotes/deb/apt/18600e5b-4914-4bbd-ae2a-bc0488c30e07/",
    "pulp_labels": {},
    "pulp_last_updated": "2021-04-30T06:39:24.047081Z",
    "rate_limit": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "sync_installer": false,
    "sync_sources": false,
    "sync_udebs": false,
    "tls_validation": true,
    "total_timeout": null,
    "url": "http://debian.mirror.digitalpacific.com.au/debian/"
}

All subsequent attempts to synchronise a repository with the remote above fail immediately with the following error:

Apr 30 06:39:46 pulphost.domain.local rq[22865]: TypeError: the JSON object must be str, bytes or bytearray, not 'list'

Actions

Copy link

Updated by simon_t over 3 years ago

Full log traceback:

Apr 30 06:39:44 pulphost.domain.local gunicorn[23232]: pulp [fb5f8febe0cf4fc88bc1857f87f4054c]:  - admin [30/Apr/2021:06:39:44 +0000] "POST /pulp/api/v3/repositories/deb/apt/dbd1bbf7-a5ef-46b6-922a-0ddf62b92ebc/sync/ HTTP/1.0" 202 67 "-" "HTTPie/0.9.8"
Apr 30 06:39:44 pulphost.domain.local rq[22865]: pulp [None]: rq.worker:INFO: 22865@pulphost.domain.local: bda07a1f-7f81-4f50-9914-ae453df2ca26
Apr 30 06:39:44 pulphost.domain.local rq[23117]: pulp [fb5f8febe0cf4fc88bc1857f87f4054c]: rq.worker:INFO: resource-manager: Job OK (02c00f37-c7a7-47d6-b5d0-ee7f09b4069e)
Apr 30 06:39:46 pulphost.domain.local rq[22865]: pulp [fb5f8febe0cf4fc88bc1857f87f4054c]: pulp_deb.app.tasks.synchronizing:INFO: Downloading Release file for distribution: "bullseye"
Apr 30 06:39:46 pulphost.domain.local rq[22865]: pulp [fb5f8febe0cf4fc88bc1857f87f4054c]: rq.worker:ERROR: Traceback (most recent call last):
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 307, in download_factory
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     return self._download_factory
Apr 30 06:39:46 pulphost.domain.local rq[22865]: AttributeError: 'AptRemote' object has no attribute '_download_factory'
Apr 30 06:39:46 pulphost.domain.local rq[22865]: During handling of the above exception, another exception occurred:
Apr 30 06:39:46 pulphost.domain.local rq[22865]: Traceback (most recent call last):
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/worker.py", line 1008, in perform_job
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     rv = job.perform()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 706, in perform
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     self._result = self._execute()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 729, in _execute
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     result = self.func(*self.args, **self.kwargs)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 122, in synchronize
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     loop.run_until_complete(pipeline)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     return future.result()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await asyncio.gather(*futures)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await self.run()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 174, in run
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     pb.done += task.result()  # download_count
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 200, in _handle_content_unit
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await asyncio.gather(*downloaders_for_content)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 135, in download
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await super().download()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 87, in download
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     downloader = self.remote.get_downloader(url=self.url, **validation_kwargs)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 371, in get_downloader
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     return self.download_factory.build(url, **kwargs)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 309, in download_factory
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     self._download_factory = DownloaderFactory(self)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/factory.py", line 88, in __init__
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     self._session = self._make_aiohttp_session_from_remote()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/factory.py", line 129, in _make_aiohttp_session_from_remote
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     headers.update(json.loads(self._remote.headers))
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/lib/python3.6/json/__init__.py", line 348, in loads
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     'not {!r}'.format(s.__class__.__name__))
Apr 30 06:39:46 pulphost.domain.local rq[22865]: TypeError: the JSON object must be str, bytes or bytearray, not 'list'
Apr 30 06:39:46 pulphost.domain.local rq[22865]: Traceback (most recent call last):
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 307, in download_factory
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     return self._download_factory
Apr 30 06:39:46 pulphost.domain.local rq[22865]: AttributeError: 'AptRemote' object has no attribute '_download_factory'
Apr 30 06:39:46 pulphost.domain.local rq[22865]: During handling of the above exception, another exception occurred:
Apr 30 06:39:46 pulphost.domain.local rq[22865]: Traceback (most recent call last):
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/worker.py", line 1008, in perform_job
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     rv = job.perform()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 706, in perform
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     self._result = self._execute()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 729, in _execute
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     result = self.func(*self.args, **self.kwargs)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 122, in synchronize
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     loop.run_until_complete(pipeline)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     return future.result()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await asyncio.gather(*futures)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await self.run()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 174, in run
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     pb.done += task.result()  # download_count
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 200, in _handle_content_unit
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await asyncio.gather(*downloaders_for_content)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 135, in download
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     await super().download()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 87, in download
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     downloader = self.remote.get_downloader(url=self.url, **validation_kwargs)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 371, in get_downloader
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     return self.download_factory.build(url, **kwargs)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 309, in download_factory
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     self._download_factory = DownloaderFactory(self)
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/factory.py", line 88, in __init__
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     self._session = self._make_aiohttp_session_from_remote()
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/factory.py", line 129, in _make_aiohttp_session_from_remote
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     headers.update(json.loads(self._remote.headers))
Apr 30 06:39:46 pulphost.domain.local rq[22865]:   File "/usr/lib/python3.6/json/__init__.py", line 348, in loads
Apr 30 06:39:46 pulphost.domain.local rq[22865]:     'not {!r}'.format(s.__class__.__name__))
Apr 30 06:39:46 pulphost.domain.local rq[22865]: TypeError: the JSON object must be str, bytes or bytearray, not 'list'

Actions

Copy link

Updated by quba42 over 3 years ago

I think this part of the remote and API are entirely inherited from pulpcore, so this might actually be a pulpcore issue. (Or else an issue with how pulp_deb interacts with pulpcore).

I will try to ask around for people who know more about it then myself.

Actions

Copy link

Updated by bmbouter over 3 years ago

Project changed from Debian Support to Pulp

Moving to pulpcore since the issue very likely in the DownloaderFactory area.

Actions

Copy link

Updated by ipanova@redhat.com over 3 years ago

The source of the problem seems to come from the wrong data format https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L304 https://github.com/pulp/pulpcore/blob/master/pulpcore/app/serializers/repository.py#L171

Actions

Copy link

Updated by dkliban@redhat.com over 3 years ago

Status changed from NEW to ASSIGNED
Assignee set to dkliban@redhat.com
Sprint set to Sprint 96

Actions

Copy link

Updated by pulpbot over 3 years ago

Status changed from ASSIGNED to POST

PR: https://github.com/pulp/pulpcore/pull/1323

Actions

Copy link

Updated by fao89 over 3 years ago

Triaged changed from No to Yes
Sprint changed from Sprint 96 to Sprint 97

Added by dkliban@redhat.com over 3 years ago

Revision 22993ffa | View on GitHub

Fixes ability to specify custom headers on a Remote

The Remotes APIs accepts a list of headers to be sent with each request during sync. However, the headers were not being properly passed to aiohttp. This patch fixes how the headers are passed to aiohttp.

This patch also ensures that any User-Agent headers configured on a remote are appended to the User-Agent string generated by pulp.

fixes: #8689 https://pulp.plan.io/issues/8689

Actions

Copy link