Issue #7651
closed
Pulp Error: Network is unreachable during RepositoriesAnsibleApi.sync
Description
Attempting a sync via the pulp-ansible plugin, the following error is thrown:
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
rv = job.perform()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
self._result = self._execute()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
return self.func(*self.args, **self.kwargs)
File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 81, in sync
d_version.create()
File "/usr/local/lib/pulp/src/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 148, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/pulp/src/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/pulp/src/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 288, in run
async for metadata in self._fetch_collections():
File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 362, in _fetch_collections
initial_data = parse_metadata(await downloader.run())
File "/usr/local/lib/pulp/src/pulpcore/pulpcore/download/base.py", line 227, in run
return await self._run(extra_data=extra_data)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/downloaders.py", line 88, in _run
return await super()._run(extra_data=extra_data)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/pulp/src/pulpcore/pulpcore/download/http.py", line 209, in _run
async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
self._resp = await self._coro
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 483, in _request
timeout=real_timeout
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
req, traces, timeout)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
raise last_exc
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
req=req, client_error=client_error)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
description: Cannot connect to host galaxy-dev.ansible.com:443 ssl:default [Network
is unreachable]
The remote url provided was https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content These are the versions present:
curl https://`hostname`/pulp/api/v3/status/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1397 100 1397 0 0 7492 0 --:--:-- --:--:-- --:--:-- 7551
{
"versions": [
{
"component": "pulpcore",
"version": "3.7.1"
},
{
"component": "pulp_2to3_migration",
"version": "0.4.0.dev0"
},
{
"component": "pulp_file",
"version": "1.4.0.dev0"
},
{
"component": "pulp_container",
"version": "2.2.0.dev0"
},
{
"component": "pulp_ansible",
"version": "0.4.1"
},
{
"component": "pulp_certguard",
"version": "1.1.0.dev0"
},
{
"component": "pulp_rpm",
"version": "3.7.0"
}
],
"online_workers": [
{
"pulp_created": "2020-10-02T17:45:43.215436Z",
"pulp_href": "/pulp/api/v3/workers/974769cc-b099-469e-8694-4c73e56fa7f7/",
"name": "676@centos7-proxy-devel.jjeffers.example.com",
"last_heartbeat": "2020-10-05T16:13:36.733971Z"
},
{
"pulp_created": "2020-09-30T17:11:40.333349Z",
"pulp_href": "/pulp/api/v3/workers/4ad803de-500a-4061-9092-5dcd9f212b5e/",
"name": "resource-manager",
"last_heartbeat": "2020-10-05T16:13:38.437627Z"
},
{
"pulp_created": "2020-10-02T17:45:41.854532Z",
"pulp_href": "/pulp/api/v3/workers/6a7edf97-736d-49b6-9c28-dcbd12f51380/",
"name": "695@centos7-proxy-devel.jjeffers.example.com",
"last_heartbeat": "2020-10-05T16:13:38.636631Z"
}
],
"online_content_apps": [
{
"name": "1195@centos7-proxy-devel.jjeffers.example.com",
"last_heartbeat": "2020-10-05T16:13:37.690219Z"
},
{
"name": "1210@centos7-proxy-devel.jjeffers.example.com",
"last_heartbeat": "2020-10-05T16:13:37.691302Z"
}
],
"database_connection": {
"connected": true
},
"redis_connection": {
"connected": true
},
"storage": {
"total": 42927656960,
"used": 6078078976,
"free": 36849577984
}
}
Relevant extract from katello server log showing API call with RespositorySyncURL parameters:
10:38:20 rails.1 | 2020-10-05T10:38:20 [D|kat|] Calling API: RepositoriesAnsibleApi.sync ...
10:38:20 rails.1 | 2020-10-05T10:38:20 [D|kat|] HTTP request body param ~BEGIN~
10:38:20 rails.1 | | {"remote":"/pulp/api/v3/remotes/ansible/collection/62b69866-9652-4243-a001-cb96c9cf3a10/","mirror":true}
Using cURL from the same instance that hosts the pulpserver:
[vagrant@centos7-proxy-devel ~]$ curl -i -X GET https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content
HTTP/1.1 301 Moved Permanently
Date: Mon, 05 Oct 2020 16:17:20 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d1e1ffeb69079587cc03db60aa3a4eb6a1601914640; expires=Wed, 04-Nov-20 16:17:20 GMT; path=/; domain=.ansible.com; HttpOnly; SameSite=Lax
Location: /api/v2/collections/testing/ansible_testing_content/
X-REQUEST-ID: e727e193a65f41ddb23696203369edbf
Set-Cookie: af91cb160f5f76cd3bc226a05339849a=1aabc03ebf238aabd8c18f839b624077; path=/; HttpOnly; Secure
Cache-control: private
CF-Cache-Status: DYNAMIC
cf-request-id: 059b26b096000007fa3a278200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5dd873c75c3107fa-ATL
Updated by fao89 over 2 years ago
it seems to be a problem with ipv6: https://stackoverflow.com/a/48008873 Maybe this needs to be addressed at pulpcore: https://github.com/pulp/pulpcore/blob/master/pulpcore/download/http.py#L145
Note: I couldn't reproduce the error on both: fedora31 and centos7 box
(pulp) [vagrant@pulp3-source-centos7 pulp_ansible]$ pip list | grep aio
aiodns 2.0.0
aiofiles 0.5.0
aiohttp 3.6.2
Updated by bmbouter over 2 years ago
fao89 wrote:
it seems to be a problem with ipv6: https://stackoverflow.com/a/48008873 Maybe this needs to be addressed at pulpcore: https://github.com/pulp/pulpcore/blob/master/pulpcore/download/http.py#L145
Note: I couldn't reproduce the error on both: fedora31 and centos7 box
(pulp) [vagrant@pulp3-source-centos7 pulp_ansible]$ pip list | grep aio aiodns 2.0.0 aiofiles 0.5.0 aiohttp 3.6.2
Before we change the aiohttp resolver I think we should confirm that this wasn't an environmental issue. For example in the past when users have VPN's running and they stop routing packets clients raise NetworkUnreachable errors just like this one. I recommend we have a reproducer before we make any code changes related to this. Is a reproducer available?
Updated by jdjeffers over 2 years ago
Might be helpful, from that proxy host:
[vagrant@centos7-proxy-devel src]$ telnet galaxy-dev.ansible.com 80
Trying 104.26.0.250...
Connected to galaxy-dev.ansible.com.
Escape character is '^]'.
^]
Updated by jdjeffers over 2 years ago
bmbouter wrote:
I recommend we have a reproducer before we make any code changes related to this. Is a reproducer available?
I'm working to get a reproducer environment available.
Updated by bmbouter over 2 years ago
jdjeffers wrote:
bmbouter wrote:
I recommend we have a reproducer before we make any code changes related to this. Is a reproducer available?
I'm working to get a reproducer environment available.
Perfect! We're happy to debug in that environment.
Updated by jdjeffers over 2 years ago
This appears to be a networking issue with the proxy itself, and is probably not specific to pulp_ansible. I'll leave the environment up for a while in case anyone would like to explore. But we can close this issue.
Updated by bmbouter over 2 years ago
- Status changed from NEW to CLOSED - WORKSFORME
OK great, thank you for the confirmation. I'll close for now, and let us know if there is anything we should do.
.