Project

Profile

Help

Issue #7651

Pulp Error: Network is unreachable during RepositoriesAnsibleApi.sync

Added by jdjeffers 2 months ago. Updated about 2 months ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Attempting a sync via the pulp-ansible plugin, the following error is thrown:



        File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
                  rv = job.perform()
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
                  self._result = self._execute()
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
                  return self.func(*self.args, **self.kwargs)
                File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 81, in sync
                  d_version.create()
                File "/usr/local/lib/pulp/src/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 148, in create
                  loop.run_until_complete(pipeline)
                File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
                  return future.result()
                File "/usr/local/lib/pulp/src/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
                  await asyncio.gather(*futures)
                File "/usr/local/lib/pulp/src/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
                  await self.run()
                File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 288, in run
                  async for metadata in self._fetch_collections():
                File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 362, in _fetch_collections
                  initial_data = parse_metadata(await downloader.run())
                File "/usr/local/lib/pulp/src/pulpcore/pulpcore/download/base.py", line 227, in run
                  return await self._run(extra_data=extra_data)
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
                  ret = await target(*args, **kwargs)
                File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/downloaders.py", line 88, in _run
                  return await super()._run(extra_data=extra_data)
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
                  ret = await target(*args, **kwargs)
                File "/usr/local/lib/pulp/src/pulpcore/pulpcore/download/http.py", line 209, in _run
                  async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
                  self._resp = await self._coro
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 483, in _request
                  timeout=real_timeout
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
                  proto = await self._create_connection(req, traces, timeout)
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
                  req, traces, timeout)
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
                  raise last_exc
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
                  req=req, client_error=client_error)
                File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
                  raise client_error(req.connection_key, exc) from exc
            description: Cannot connect to host galaxy-dev.ansible.com:443 ssl:default [Network
              is unreachable]

The remote url provided was https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content These are the versions present:

curl https://`hostname`/pulp/api/v3/status/   --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1397  100  1397    0     0   7492      0 --:--:-- --:--:-- --:--:--  7551
{
  "versions": [
    {
      "component": "pulpcore",
      "version": "3.7.1"
    },
    {
      "component": "pulp_2to3_migration",
      "version": "0.4.0.dev0"
    },
    {
      "component": "pulp_file",
      "version": "1.4.0.dev0"
    },
    {
      "component": "pulp_container",
      "version": "2.2.0.dev0"
    },
    {
      "component": "pulp_ansible",
      "version": "0.4.1"
    },
    {
      "component": "pulp_certguard",
      "version": "1.1.0.dev0"
    },
    {
      "component": "pulp_rpm",
      "version": "3.7.0"
    }
  ],
  "online_workers": [
    {
      "pulp_created": "2020-10-02T17:45:43.215436Z",
      "pulp_href": "/pulp/api/v3/workers/974769cc-b099-469e-8694-4c73e56fa7f7/",
      "name": "676@centos7-proxy-devel.jjeffers.example.com",
      "last_heartbeat": "2020-10-05T16:13:36.733971Z"
    },
    {
      "pulp_created": "2020-09-30T17:11:40.333349Z",
      "pulp_href": "/pulp/api/v3/workers/4ad803de-500a-4061-9092-5dcd9f212b5e/",
      "name": "resource-manager",
      "last_heartbeat": "2020-10-05T16:13:38.437627Z"
    },
    {
      "pulp_created": "2020-10-02T17:45:41.854532Z",
      "pulp_href": "/pulp/api/v3/workers/6a7edf97-736d-49b6-9c28-dcbd12f51380/",
      "name": "695@centos7-proxy-devel.jjeffers.example.com",
      "last_heartbeat": "2020-10-05T16:13:38.636631Z"
    }
  ],
  "online_content_apps": [
    {
      "name": "1195@centos7-proxy-devel.jjeffers.example.com",
      "last_heartbeat": "2020-10-05T16:13:37.690219Z"
    },
    {
      "name": "1210@centos7-proxy-devel.jjeffers.example.com",
      "last_heartbeat": "2020-10-05T16:13:37.691302Z"
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": true
  },
  "storage": {
    "total": 42927656960,
    "used": 6078078976,
    "free": 36849577984
  }
}

Relevant extract from katello server log showing API call with RespositorySyncURL parameters:

10:38:20 rails.1   | 2020-10-05T10:38:20 [D|kat|] Calling API: RepositoriesAnsibleApi.sync ...
10:38:20 rails.1   | 2020-10-05T10:38:20 [D|kat|] HTTP request body param ~BEGIN~
10:38:20 rails.1   |  | {"remote":"/pulp/api/v3/remotes/ansible/collection/62b69866-9652-4243-a001-cb96c9cf3a10/","mirror":true}

Using cURL from the same instance that hosts the pulpserver:

[vagrant@centos7-proxy-devel ~]$ curl -i -X GET https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content
HTTP/1.1 301 Moved Permanently
Date: Mon, 05 Oct 2020 16:17:20 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d1e1ffeb69079587cc03db60aa3a4eb6a1601914640; expires=Wed, 04-Nov-20 16:17:20 GMT; path=/; domain=.ansible.com; HttpOnly; SameSite=Lax
Location: /api/v2/collections/testing/ansible_testing_content/
X-REQUEST-ID: e727e193a65f41ddb23696203369edbf
Set-Cookie: af91cb160f5f76cd3bc226a05339849a=1aabc03ebf238aabd8c18f839b624077; path=/; HttpOnly; Secure
Cache-control: private
CF-Cache-Status: DYNAMIC
cf-request-id: 059b26b096000007fa3a278200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5dd873c75c3107fa-ATL

History

#1 Updated by fao89 2 months ago

it seems to be a problem with ipv6: https://stackoverflow.com/a/48008873 Maybe this needs to be addressed at pulpcore: https://github.com/pulp/pulpcore/blob/master/pulpcore/download/http.py#L145

Note: I couldn't reproduce the error on both: fedora31 and centos7 box

(pulp) [vagrant@pulp3-source-centos7 pulp_ansible]$ pip list | grep aio
aiodns                          2.0.0
aiofiles                        0.5.0
aiohttp                         3.6.2

#2 Updated by bmbouter about 2 months ago

fao89 wrote:

it seems to be a problem with ipv6: https://stackoverflow.com/a/48008873 Maybe this needs to be addressed at pulpcore: https://github.com/pulp/pulpcore/blob/master/pulpcore/download/http.py#L145

Note: I couldn't reproduce the error on both: fedora31 and centos7 box

(pulp) [vagrant@pulp3-source-centos7 pulp_ansible]$ pip list | grep aio
aiodns                          2.0.0
aiofiles                        0.5.0
aiohttp                         3.6.2

Before we change the aiohttp resolver I think we should confirm that this wasn't an environmental issue. For example in the past when users have VPN's running and they stop routing packets clients raise NetworkUnreachable errors just like this one. I recommend we have a reproducer before we make any code changes related to this. Is a reproducer available?

#3 Updated by jdjeffers about 2 months ago

Might be helpful, from that proxy host:

[vagrant@centos7-proxy-devel src]$ telnet galaxy-dev.ansible.com 80
Trying 104.26.0.250...
Connected to galaxy-dev.ansible.com.
Escape character is '^]'.
^]

#4 Updated by jdjeffers about 2 months ago

bmbouter wrote:

I recommend we have a reproducer before we make any code changes related to this. Is a reproducer available?

I'm working to get a reproducer environment available.

#5 Updated by bmbouter about 2 months ago

jdjeffers wrote:

bmbouter wrote:

I recommend we have a reproducer before we make any code changes related to this. Is a reproducer available?

I'm working to get a reproducer environment available.

Perfect! We're happy to debug in that environment.

#6 Updated by jdjeffers about 2 months ago

This appears to be a networking issue with the proxy itself, and is probably not specific to pulp_ansible. I'll leave the environment up for a while in case anyone would like to explore. But we can close this issue.

#7 Updated by bmbouter about 2 months ago

  • Status changed from NEW to CLOSED - WORKSFORME

OK great, thank you for the confirmation. I'll close for now, and let us know if there is anything we should do.

Please register to edit this issue

Also available in: Atom PDF