Issue #7205: ClientConnectorSSLError during remote sync with cdn.redhat.com - Pulp

Actions

Send by e-mail Copy link

Issue #7205

closed

ClientConnectorSSLError during remote sync with cdn.redhat.com

Added by pc over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - NOTABUG

Priority:

Normal

Assignee:

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

3.3.0

Platform Release:

OS:

CentOS 7

Triaged:

Groomed:

Sprint Candidate:

Tags:

Sprint:

Quarter:

Description

Steps:

Install Pulp 3.4.1 or 3.3.0 (both exhibit) using pulp_installer Ansible playbook on CentOS 7.7 x86_64.
Configure new RPM Repository and RPM Remote
Run RPM Remote Sync
Check task status, observe running states of Downloading Metadata Files, Downloading Artifacts, and Associating Content, Parsed Packages.
Check task status after ~60 seconds, observe failed state of Downloading Artifacts.
Run RPM Remote Sync repeatedly (~6 times) and eventually all packages are downloaded and task completes successfully.

Notes:

Reproduced on two different hosts in completely separate network infra.
No issues observed with non-Pulp package fetching. Successful runs of wget and high-concurrency aira2c download of repo contents.
iptraf-ng's IP Traffic Monitor shows much higher number of lingering connections for Pulp than wget or aria2c.

Traceback of failed task, unfurled:

  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
    rv = job.perform()
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
    self._result = self._execute()
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
    dv.create()
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
    await asyncio.gather(*futures)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
    pb.done += task.result()  # download_count
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 157, in _handle_content_unit
    await asyncio.gather(*downloaders_for_content)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 81, in download
    download_result = await downloader.run(extra_data=self.extra_data)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py", line 221, in run
    return await self._run(extra_data=extra_data)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 184, in _run
    async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
    req, traces, timeout)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
    req=req, client_error=client_error)
  File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc

$ http -a admin:$PULP_PASS http://0/pulp/api/v3/tasks/731fd427-fc27-4f2a-bb95-8dab519be94b/

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 167, in synchronize\n    dv.create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n    pb.done += task.result()  # download_count\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 81, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 184, in _run\n    async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n    self._resp = await self._coro\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 483, in _request\n    timeout=real_timeout\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 523, in connect\n    proto = await self._create_connection(req, traces, timeout)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 859, in _create_connection\n    req, traces, timeout)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 1004, in _create_direct_connection\n    raise last_exc\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 986, in _create_direct_connection\n    req=req, client_error=client_error)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/connector.py\", line 941, in _wrap_create_connection\n    raise ClientConnectorSSLError(req.connection_key, exc) from exc\n"
    },
    "finished_at": "2020-07-23T10:45:23.765267Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "downloading.artifacts",
            "done": 511,
            "message": "Downloading Artifacts",
            "state": "failed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.packages",
            "done": 3004,
            "message": "Parsed Packages",
            "state": "canceled",
            "suffix": null,
            "total": 17914
        },
        {
            "code": "downloading.metadata",
            "done": 5,
            "message": "Downloading Metadata Files",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 2501,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.comps",
            "done": 3,
            "message": "Parsed Comps",
            "state": "completed",
            "suffix": null,
            "total": 3
        },
        {
            "code": "parsing.advisories",
            "done": 2372,
            "message": "Parsed Advisories",
            "state": "completed",
            "suffix": null,
            "total": 2372
        }
    ],
    "pulp_created": "2020-07-23T10:44:23.145571Z",
    "pulp_href": "/pulp/api/v3/tasks/731fd427-fc27-4f2a-bb95-8dab519be94b/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/",
        "/pulp/api/v3/remotes/rpm/rpm/3bb3d1c0-39b4-4569-b1f0-87601196de6a/"
    ],
    "started_at": "2020-07-23T10:44:23.272211Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/ecd83320-435f-464c-86cb-7627efe1c1e2/"
}

$ http -a admin:$PULP_PASS http://0/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/

{
    "description": null,
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/versions/0/",
    "metadata_signing_service": null,
    "name": "content-dist-rhel-server-7-7.6-x86_64-optional-os",
    "pulp_created": "2020-07-23T10:03:12.350952Z",
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/e0d99da4-8db0-48ee-97f1-375db652644b/versions/"
}

$ http -a admin:$PULP_PASS http://0/pulp/api/v3/remotes/rpm/rpm/3bb3d1c0-39b4-4569-b1f0-87601196de6a/

{
    "ca_cert": "c77dce7447d4862f2b5fbd794cf6fcdc6f01e3c313cc9a12d2bf47cfd6b8db21",
    "client_cert": "<redacted>",
    "client_key": "<redacted>",
    "download_concurrency": 20,
    "name": "content-dist-rhel-server-7-7.6-x86_64-optional-os",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2020-07-23T10:03:13.610779Z",
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/3bb3d1c0-39b4-4569-b1f0-87601196de6a/",
    "pulp_last_updated": "2020-07-23T10:23:34.386392Z",
    "tls_validation": false,
    "url": "https://cdn.redhat.com/content/dist/rhel/server/7/7.6/x86_64/optional/os"
}

Actions

Copy link

Updated by bmbouter over 3 years ago

It's possible Pulp is forming too many parallel connections to servers, they get angry, and close TCP connections non-gracefully causing errors like these. Pulp Remotes have a download_concurrency options, which defaults to 20. Can you try to set this value on your remote and reproduce this issue. Try values like to 10, 5, or 3 to test if that makes these errors go away. The default is 20 which is likely too high. We're considering adjusting it, so let us know what works well for you please, or if it doesn't resolve it.

https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create

Actions

Copy link