Project

Profile

Help

Issue #7321

pulp_rpm: Cannot synchronize repos behind proxy [Network is unreachable]

Added by Anonymous 2 months ago. Updated about 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
08/14/2020
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 81
Quarter:

Description

I had set up pulpcore and pulp_rpm and had working repos that I had synced successfully.

It stopped working when I upgraded pulp_core from v3.4.2 to v3.5.0.

Interestingly the error message does not mention the remote url the way it is in the remote object. E.g. for this remote:

{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/dcebc720-cec0-440d-8327-605ebefef778/",
  "pulp_created": "2020-08-14T19:30:59.442292Z",
  "name": "test",
  "url": "https://fixtures.pulpproject.org/rpm-unsigned/",
  "ca_cert": null,
  "client_cert": null,
  "client_key": null,
  "tls_validation": true,
  "proxy_url": http://<my own proxy>,
  "username": null,
  "password": null,
  "pulp_last_updated": "2020-08-14T19:30:59.442315Z",
  "download_concurrency": 20,
  "policy": "immediate",
  "sles_auth_token": null
}
Aug 14 21:49:42 chsss039 rq[303140]: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host fixtures.pulpproject.org:443 ssl:default [Connect call failed ('8.43.85.236', 443)]
Aug 14 21:49:42 chsss039 rq[303140]: Traceback (most recent call last):
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
  return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
File "/usr/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection
  raise exceptions[0]
File "/usr/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection
  yield from self.sock_connect(sock, address)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect
  return (yield from fut)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb
  raise OSError(err, 'Connect call failed %s' % (address,))
Aug 14 21:49:42 chsss039 rq[303140]: TimeoutError: [Errno 110] Connect call failed ('8.43.85.236', 443)
Aug 14 21:49:42 chsss039 rq[303140]: The above exception was the direct cause of the following exception:
Aug 14 21:49:42 chsss039 rq[303140]: Traceback (most recent call last):
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
  rv = job.perform()
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
  self._result = self._execute()
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
  return self.func(*self.args, **self.kwargs)
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 211, in synchronize
  remote_url = fetch_remote_url(remote)
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 136, in fetch_remote_url
  downloader.fetch()
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/pulpcore/download/base.py", line 160, in fetch
  return done.pop().result()
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/pulpcore/download/base.py", line 227, in run
  return await self._run(extra_data=extra_data)
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 41, in _run
  async with self.session.get(url) as response:
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
  self._resp = await self._coro
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/client.py", line 483, in _request
  timeout=real_timeout
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
  proto = await self._create_connection(req, traces, timeout)
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
  req, traces, timeout)
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
  raise last_exc
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
  req=req, client_error=client_error)
File "/var/lib/pulp/pulpvenv/lib64/python3.6/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
  raise client_error(req.connection_key, exc) from exc

It seems to be related to the proxy settings. When I try to synchronize I cannot see the traffic on the proxy server.

How to reproduce:

  • On server where pulp is running block outgoing connection (e.g. ip of repo) to repository.
  • Try to sync remote with proxy_url

Associated revisions

Revision c717196d View on GitHub
Added by ppicka about 2 months ago

Sync through proxy fix

Fix sync using proxy server. Custom downloader doesn't passed proxy info.

[nocoverage]

closes: #7321 https://pulp.plan.io/issues/7321

Revision c58fbacd View on GitHub
Added by ppicka about 2 months ago

Sync through proxy fix

Fix sync using proxy server. Custom downloader doesn't passed proxy info.

[nocoverage]

closes: #7321 https://pulp.plan.io/issues/7321 (cherry picked from commit c717196dab597f4a21ed71e3ce10097d14dc49f3)

History

#1 Updated by Anonymous 2 months ago

Sorry, there was a typo above. "It stopped working when I upgraded pulp_core pulp_rpm from v3.4.2 to v3.5.0."

However, the issue persists even after updating both pulpcore and pulp_rpm to 3.6.0.

#2 Updated by fao89 2 months ago

  • Sprint set to Sprint 79

#3 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 79 to Sprint 80

#4 Updated by ppicka about 2 months ago

  • Triaged changed from No to Yes
  • Groomed changed from No to Yes

#5 Updated by jsherril@redhat.com about 2 months ago

  • Tags Katello added

#6 Updated by ppicka about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ppicka

#7 Updated by ppicka about 2 months ago

Tested on Centos and Fedora, with version 3.4.2, 3.5 and 3.6 but I am unable to reproduce with local (same machine) or remote proxy server. Can you please specify the reproducer? More information about proxy server (software, its config, location[infra]) could be helpful too.

#8 Updated by binlinf0 about 2 months ago

We are experiencing the same issue after upgrading from 3.3 to 3.6..1 on both rhel 7.6 and 7.7. We are using a remote proxy with a self signed certificate. We will provide more information on proxy once we receive it from our network team.

"description": "Cannot connect to host cdn.redhat.com:443 ssl:default 

[Connect call failed ('173.222.100.251', 443)]", "traceback": " File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 884, in perform_job\n rv = job.perform()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform\n self._result = self._execute()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/sy nchronizing.py", line 211, in synchronize\n remote_url = fetch_remote_url(remote)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tas ks/synchronizing.py", line 136, in fetch_remote_url\n downloader.fetch()\n
File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 160, in fetch\n return done.pop().result()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/ download/base.py", line 227, in run\n return await self._run(extra_data=extra_data)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/dow nloaders.py", line 87, in _run\n async with self.session.get(url) as response:\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.p y", line 1012, in aenter\n self._resp = await self._coro\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.p y", line 483, in _request\n timeout=real_timeout\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connecto r.py", line 523, in connect\n proto = await self._create_connection(req, traces, timeout)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connecto r.py", line 859, in _create_connection\n req, traces, timeout)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connecto r.py", line 1004, in _create_direct_connection\n raise last_exc\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connecto r.py", line 986, in _create_direct_connection\n req=req, client_error=client_error)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connecto r.py", line 943, in _wrap_create_connection\n raise client_error(req.connection_key, exc) from exc\n"

The pulp version we are running: "versions": [ { "component": "pulpcore", "version": "3.6.0" }, { "component": "pulp_rpm", "version": "3.6.1" }, { "component": "pulp_file", "version": "1.2.0" }

#9 Updated by binlinf0 about 2 months ago

Here is response from our network team. It looks like pulp is trying to connect "localhostpulp" "Looking at the proxy logs it looks like you are connecting to

cs_uri_scheme=http cs_host=localhostpulp cs_uri_port=80 cs_uri_path=/api/v3/remotes/rpm/rpm/fd4fb91f-c8ca-4c8d-a88b-880d88897a8e

They proxy is denying this as we can't categorize this hostname. Even if we allowed it...the proxy can't resolve this to an IP address.

I'm guessing the package gets loaded into local webserver. If this is indeed a localhost name listed in the hostfile...it should not be using a proxy for this portion of traffic"

#10 Updated by ppicka about 2 months ago

  • Description updated (diff)

#11 Updated by pulpbot about 2 months ago

  • Status changed from ASSIGNED to POST

#12 Updated by ppicka about 2 months ago

  • Project changed from Pulp to RPM Support
  • Category deleted (Operator)

#13 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 80 to Sprint 81

#14 Updated by ppicka about 2 months ago

  • Status changed from POST to MODIFIED

#15 Updated by fao89 about 2 months ago

  • Sprint/Milestone set to 3.6.2

#16 Updated by pulpbot about 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF