Actions
Issue #7205
closedClientConnectorSSLError during remote sync with cdn.redhat.com
Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
3.3.0
Platform Release:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Description
Steps:
- Install Pulp 3.4.1 or 3.3.0 (both exhibit) using pulp_installer Ansible playbook on CentOS 7.7 x86_64.
- Configure new RPM Repository and RPM Remote
- Run RPM Remote Sync
- Check task status, observe running states of Downloading Metadata Files, Downloading Artifacts, and Associating Content, Parsed Packages.
- Check task status after ~60 seconds, observe failed state of Downloading Artifacts.
- Run RPM Remote Sync repeatedly (~6 times) and eventually all packages are downloaded and task completes successfully.
Notes:
- Reproduced on two different hosts in completely separate network infra.
- No issues observed with non-Pulp package fetching. Successful runs of wget and high-concurrency aira2c download of repo contents.
- iptraf-ng's IP Traffic Monitor shows much higher number of lingering connections for Pulp than wget or aria2c.
Traceback of failed task, unfurled:
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
rv = job.perform()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
self._result = self._execute()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
return self.func(*self.args, **self.kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
dv.create()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
pb.done += task.result() # download_count
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 157, in _handle_content_unit
await asyncio.gather(*downloaders_for_content)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 81, in download
download_result = await downloader.run(extra_data=self.extra_data)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py", line 221, in run
return await self._run(extra_data=extra_data)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 184, in _run
async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
self._resp = await self._coro
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 483, in _request
timeout=real_timeout
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
req, traces, timeout)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
raise last_exc
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
req=req, client_error=client_error)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
raise ClientConnectorSSLError(req.connection_key, exc) from exc
$ http -a admin:$PULP_PASS http://0/pulp/api/v3/tasks/731fd427-fc27-4f2a-bb95-8dab519be94b/
{
"child_tasks": [],
"created_resources": [],
"error": {
"description": "Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]",
"traceback": " File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 167, in synchronize\n dv.create()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n pb.done += task.result() # download_count\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 81, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n return await self._run(extra_data=extra_data)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n ret = await target(*args, **kwargs)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 184, in _run\n async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n self._resp = await self._coro\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 483, in _request\n timeout=real_timeout\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 523, in connect\n proto = await self._create_connection(req, traces, timeout)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 859, in _create_connection\n req, traces, timeout)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 1004, in _create_direct_connection\n raise last_exc\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 986, in _create_direct_connection\n req=req, client_error=client_error)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 941, in _wrap_create_connection\n raise ClientConnectorSSLError(req.connection_key, exc) from exc\n"
},
"finished_at": "2020-07-23T10:45:23.765267Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "downloading.artifacts",
"done": 511,
"message": "Downloading Artifacts",
"state": "failed",
"suffix": null,
"total": null
},
{
"code": "parsing.packages",
"done": 3004,
"message": "Parsed Packages",
"state": "canceled",
"suffix": null,
"total": 17914
},
{
"code": "downloading.metadata",
"done": 5,
"message": "Downloading Metadata Files",
"state": "canceled",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 2501,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": null
},
{
"code": "parsing.comps",
"done": 3,
"message": "Parsed Comps",
"state": "completed",
"suffix": null,
"total": 3
},
{
"code": "parsing.advisories",
"done": 2372,
"message": "Parsed Advisories",
"state": "completed",
"suffix": null,
"total": 2372
}
],
"pulp_created": "2020-07-23T10:44:23.145571Z",
"pulp_href": "/pulp/api/v3/tasks/731fd427-fc27-4f2a-bb95-8dab519be94b/",
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/",
"/pulp/api/v3/remotes/rpm/rpm/3bb3d1c0-39b4-4569-b1f0-87601196de6a/"
],
"started_at": "2020-07-23T10:44:23.272211Z",
"state": "failed",
"task_group": null,
"worker": "/pulp/api/v3/workers/ecd83320-435f-464c-86cb-7627efe1c1e2/"
}
$ http -a admin:$PULP_PASS http://0/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/
{
"description": null,
"latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/versions/0/",
"metadata_signing_service": null,
"name": "content-dist-rhel-server-7-7.6-x86_64-optional-os",
"pulp_created": "2020-07-23T10:03:12.350952Z",
"pulp_href": "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/",
"versions_href": "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/versions/"
}
$ http -a admin:$PULP_PASS http://0/pulp/api/v3/remotes/rpm/rpm/3bb3d1c0-39b4-4569-b1f0-87601196de6a/
{
"ca_cert": "c77dce7447d4862f2b5fbd794cf6fcdc6f01e3c313cc9a12d2bf47cfd6b8db21",
"client_cert": "<redacted>",
"client_key": "<redacted>",
"download_concurrency": 20,
"name": "content-dist-rhel-server-7-7.6-x86_64-optional-os",
"policy": "immediate",
"proxy_url": null,
"pulp_created": "2020-07-23T10:03:13.610779Z",
"pulp_href": "/pulp/api/v3/remotes/rpm/rpm/3bb3d1c0-39b4-4569-b1f0-87601196de6a/",
"pulp_last_updated": "2020-07-23T10:23:34.386392Z",
"tls_validation": false,
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7.6/x86_64/optional/os"
}
Actions