Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #6589

closed

Issues synching large Redhat repos

Added by mped over 4 years ago. Updated over 3 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Hi

I have found issues when synching large repos with RedHat when using Pulp 3.2 and 3.3 on RHEL7. Some examples of the repos are:

If I re kick off the download Pulp in some cases manage to finish the repo download. In the cases below the download failed again shortly afterwards. These logs are where there is where the server has direct access to the Internet. It also seems like it is random files pulp gets stuck on, becase sometimes the sync can run for a long time before it fails. Othertimes quite quickly.

In either case I seem to see:

  • A socket read timeout
  • A server disconnected error

One thing I'm not sure on is does Pulp 3 retry a connection when it is dropped? Or is it able to move onto the next file if a problem is found, and then log the files it failed to download?

Below is what happened when I tried to download from the debug repo:

 {
            "pulp_href": "/pulp/api/v3/tasks/11b53226-941a-4708-800a-58c1af19aa63/",
            "pulp_created": "2020-04-24T09:30:21.775169Z",
            "state": "failed",
            "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
            "started_at": "2020-04-24T09:30:21.902780Z",
            "finished_at": "2020-04-24T10:19:09.519158Z",
            "error": {
                "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 167, in synchronize\n    dv.create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n    pb.done += task.result()  # download_count\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 81, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 186, in _run\n    to_return = await self._handle_response(response)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 160, in _handle_response\n    chunk = await response.content.read(1048576)  # 1 megabyte\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py\", line 368, in read\n    await self._wait('read')\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py\", line 296, in _wait\n    await waiter\n",
                "description": "Timeout on reading data from socket"
            },
            "worker": "/pulp/api/v3/workers/69f2d00a-a47a-492c-8f9e-55167b1a8c7c/",
            "parent_task": null,
            "child_tasks": [],
            "task_group": null,
            "progress_reports": [
                {
                    "message": "Parsed Comps",
                    "code": "parsing.comps",
                    "state": "completed",
                    "total": 0,
                    "done": 0,
                    "suffix": null
                },
                {
                    "message": "Parsed Advisories",
                    "code": "parsing.advisories",
                    "state": "completed",
                    "total": 3611,
                    "done": 3611,
                    "suffix": null
                },
                {
                    "message": "Downloading Artifacts",
                    "code": "downloading.artifacts",
                    "state": "failed",
                    "total": null,
                    "done": 2622,
                    "suffix": null
                },
                {
                    "message": "Parsed Packages",
                    "code": "parsing.packages",
                    "state": "canceled",
                    "total": 9541,
                    "done": 5006,
                    "suffix": null
                },
                {
                    "message": "Downloading Metadata Files",
                    "code": "downloading.metadata",
                    "state": "canceled",
                    "total": null,
                    "done": 5,
                    "suffix": null
                },
                {
                    "message": "Associating Content",
                    "code": "associating.content",
                    "state": "canceled",
                    "total": null,
                    "done": 6000,
                    "suffix": null
                }
            ],
            "created_resources": [],
            "reserved_resources_record": [
                "/pulp/api/v3/remotes/rpm/rpm/a65d9b42-1289-4355-bc3a-5de4ae4c45c7/",
                "/pulp/api/v3/repositories/rpm/rpm/9fff639e-55f2-42da-8539-d1e1b2dd1d52/"
            ]
        },

Errors seen in /var/log/messages

Apr 24 11:19:09 pulpserver rq: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ServerTimeoutError: Timeout on reading data from socket
Apr 24 11:19:09 pulpserver rq: Traceback (most recent call last):
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
Apr 24 11:19:09 pulpserver rq: rv = job.perform()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
Apr 24 11:19:09 pulpserver rq: self._result = self._execute()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
Apr 24 11:19:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
Apr 24 11:19:09 pulpserver rq: dv.create()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 24 11:19:09 pulpserver rq: loop.run_until_complete(pipeline)
Apr 24 11:19:09 pulpserver rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 24 11:19:09 pulpserver rq: return future.result()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
Apr 24 11:19:09 pulpserver rq: await asyncio.gather(*futures)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 24 11:19:09 pulpserver rq: await self.run()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
Apr 24 11:19:09 pulpserver rq: pb.done += task.result()  # download_count
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 157, in _handle_content_unit
Apr 24 11:19:09 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 81, in download
Apr 24 11:19:09 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py", line 221, in run
Apr 24 11:19:09 pulpserver rq: return await self._run(extra_data=extra_data)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
Apr 24 11:19:09 pulpserver rq: ret = await target(*args, **kwargs)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 186, in _run
Apr 24 11:19:09 pulpserver rq: to_return = await self._handle_response(response)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 160, in _handle_response
Apr 24 11:19:09 pulpserver rq: chunk = await response.content.read(1048576)  # 1 megabyte
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py", line 368, in read
Apr 24 11:19:09 pulpserver rq: await self._wait('read')
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py", line 296, in _wait
Apr 24 11:19:09 pulpserver rq: await waiter
Apr 24 11:19:09 pulpserver rq: aiohttp.client_exceptions.ServerTimeoutError: Timeout on reading data from socket
Apr 24 11:19:09 pulpserver rq: Traceback (most recent call last):
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
Apr 24 11:19:09 pulpserver rq: rv = job.perform()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
Apr 24 11:19:09 pulpserver rq: self._result = self._execute()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
Apr 24 11:19:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
Apr 24 11:19:09 pulpserver rq: dv.create()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 24 11:19:09 pulpserver rq: loop.run_until_complete(pipeline)
Apr 24 11:19:09 pulpserver rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 24 11:19:09 pulpserver rq: return future.result()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
Apr 24 11:19:09 pulpserver rq: await asyncio.gather(*futures)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 24 11:19:09 pulpserver rq: await self.run()
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
Apr 24 11:19:09 pulpserver rq: pb.done += task.result()  # download_count
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 157, in _handle_content_unit
Apr 24 11:19:09 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 81, in download
Apr 24 11:19:09 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py", line 221, in run
Apr 24 11:19:09 pulpserver rq: return await self._run(extra_data=extra_data)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
Apr 24 11:19:09 pulpserver rq: ret = await target(*args, **kwargs)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 186, in _run
Apr 24 11:19:09 pulpserver rq: to_return = await self._handle_response(response)
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 160, in _handle_response
Apr 24 11:19:09 pulpserver rq: chunk = await response.content.read(1048576)  # 1 megabyte
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py", line 368, in read
Apr 24 11:19:09 pulpserver rq: await self._wait('read')
Apr 24 11:19:09 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py", line 296, in _wait
Apr 24 11:19:09 pulpserver rq: await waiter
Apr 24 11:19:09 pulpserver rq: aiohttp.client_exceptions.ServerTimeoutError: Timeout on reading data from socket
Apr 24 11:19:10 pulpserver rq: pulp: rq.worker:INFO: Cleaning registries for queue: 14665@unallocated.barefruit.co.uk
Apr 24 11:19:10 pulpserver rq: pulp: rq.worker:INFO: 14665@unallocated.barefruit.co.uk: fbd49388-425a-4816-9fe6-d8e5451f254b
Apr 24 11:19:10 pulpserver rq: pulp: rq.worker:INFO: 14665@unallocated.barefruit.co.uk: Job OK (fbd49388-425a-4816-9fe6-d8e5451f254b)

When I tried to resync I got the following:

        {
            "pulp_href": "/pulp/api/v3/tasks/75013a27-43aa-4cf1-8ba8-43554168c9e3/",
            "pulp_created": "2020-04-24T10:25:48.963795Z",
            "state": "failed",
            "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
            "started_at": "2020-04-24T10:25:49.160507Z",
            "finished_at": "2020-04-24T12:22:39.109603Z",
            "error": {
                "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 167, in synchronize\n    dv.create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n    pb.done += task.result()  # download_count\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 81, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 184, in _run\n    async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n    self._resp = await self._coro\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 504, in _request\n    await resp.start(conn)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 847, in start\n    message, payload = await self._protocol.read()  # type: ignore  # noqa\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py\", line 591, in read\n    await self._waiter\n",
                "description": ""
            },
            "worker": "/pulp/api/v3/workers/69f2d00a-a47a-492c-8f9e-55167b1a8c7c/",
            "parent_task": null,
            "child_tasks": [],
            "task_group": null,
            "progress_reports": [
                {
                    "message": "Parsed Comps",
                    "code": "parsing.comps",
                    "state": "completed",
                    "total": 0,
                    "done": 0,
                    "suffix": null
                },
                {
                    "message": "Parsed Advisories",
                    "code": "parsing.advisories",
                    "state": "completed",
                    "total": 3611,
                    "done": 3611,
                    "suffix": null
                },
                {
                    "message": "Parsed Packages",
                    "code": "parsing.packages",
                    "state": "completed",
                    "total": 9541,
                    "done": 9541,
                    "suffix": null
                },
                {
                    "message": "Downloading Metadata Files",
                    "code": "downloading.metadata",
                    "state": "completed",
                    "total": null,
                    "done": 5,
                    "suffix": null
                },
                {
                    "message": "Downloading Artifacts",
                    "code": "downloading.artifacts",
                    "state": "failed",
                    "total": null,
                    "done": 6500,
                    "suffix": null
                },
                {
                    "message": "Associating Content",
                    "code": "associating.content",
                    "state": "canceled",
                    "total": null,
                    "done": 12500,
                    "suffix": null
                }
            ],
            "created_resources": [],
            "reserved_resources_record": [
                "/pulp/api/v3/remotes/rpm/rpm/a65d9b42-1289-4355-bc3a-5de4ae4c45c7/",
                "/pulp/api/v3/repositories/rpm/rpm/9fff639e-55f2-42da-8539-d1e1b2dd1d52/"
            ]
        },

The output from /var/log/messages

Apr 24 13:11:17 pulpserver rq: pulp: rq.worker:INFO: Cleaning registries for queue: resource-manager
Apr 24 13:13:18 pulpserver rq: pulp: rq.worker:INFO: Cleaning registries for queue: 14711@unallocated.barefruit.co.uk
Apr 24 13:22:39 pulpserver rq: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ServerDisconnectedError
Apr 24 13:22:39 pulpserver rq: Traceback (most recent call last):
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
Apr 24 13:22:39 pulpserver rq: rv = job.perform()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
Apr 24 13:22:39 pulpserver rq: self._result = self._execute()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
Apr 24 13:22:39 pulpserver rq: return self.func(*self.args, **self.kwargs)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
Apr 24 13:22:39 pulpserver rq: dv.create()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 24 13:22:39 pulpserver rq: loop.run_until_complete(pipeline)
Apr 24 13:22:39 pulpserver rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 24 13:22:39 pulpserver rq: return future.result()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
Apr 24 13:22:39 pulpserver rq: await asyncio.gather(*futures)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 24 13:22:39 pulpserver rq: await self.run()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
Apr 24 13:22:39 pulpserver rq: pb.done += task.result()  # download_count
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 157, in _handle_content_unit
Apr 24 13:22:39 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 81, in download
Apr 24 13:22:39 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py", line 221, in run
Apr 24 13:22:39 pulpserver rq: return await self._run(extra_data=extra_data)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
Apr 24 13:22:39 pulpserver rq: ret = await target(*args, **kwargs)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 184, in _run
Apr 24 13:22:39 pulpserver rq: async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
Apr 24 13:22:39 pulpserver rq: self._resp = await self._coro
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 504, in _request
Apr 24 13:22:39 pulpserver rq: await resp.start(conn)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 847, in start
Apr 24 13:22:39 pulpserver rq: message, payload = await self._protocol.read()  # type: ignore  # noqa
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py", line 591, in read
Apr 24 13:22:39 pulpserver rq: await self._waiter
Apr 24 13:22:39 pulpserver rq: aiohttp.client_exceptions.ServerDisconnectedError
Apr 24 13:22:39 pulpserver rq: Traceback (most recent call last):
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 886, in perform_job
Apr 24 13:22:39 pulpserver rq: rv = job.perform()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
Apr 24 13:22:39 pulpserver rq: self._result = self._execute()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
Apr 24 13:22:39 pulpserver rq: return self.func(*self.args, **self.kwargs)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
Apr 24 13:22:39 pulpserver rq: dv.create()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 24 13:22:39 pulpserver rq: loop.run_until_complete(pipeline)
Apr 24 13:22:39 pulpserver rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 24 13:22:39 pulpserver rq: return future.result()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
Apr 24 13:22:39 pulpserver rq: await asyncio.gather(*futures)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 24 13:22:39 pulpserver rq: await self.run()
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
Apr 24 13:22:39 pulpserver rq: pb.done += task.result()  # download_count
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 157, in _handle_content_unit
Apr 24 13:22:39 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 81, in download
Apr 24 13:22:39 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py", line 221, in run
Apr 24 13:22:39 pulpserver rq: return await self._run(extra_data=extra_data)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 133, in retry
Apr 24 13:22:39 pulpserver rq: ret = await target(*args, **kwargs)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py", line 184, in _run
Apr 24 13:22:39 pulpserver rq: async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
Apr 24 13:22:39 pulpserver rq: self._resp = await self._coro
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py", line 504, in _request
Apr 24 13:22:39 pulpserver rq: await resp.start(conn)
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 847, in start
Apr 24 13:22:39 pulpserver rq: message, payload = await self._protocol.read()  # type: ignore  # noqa
Apr 24 13:22:39 pulpserver rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py", line 591, in read
Apr 24 13:22:39 pulpserver rq: await self._waiter
Apr 24 13:22:39 pulpserver rq: aiohttp.client_exceptions.ServerDisconnectedError

For the OS repos I saw the following:

        {
            "pulp_href": "/pulp/api/v3/tasks/6639b9ee-3f64-4c8e-848c-4c9cff779b1c/",
            "pulp_created": "2020-04-24T09:30:08.033791Z",
            "state": "failed",
            "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
            "started_at": "2020-04-24T09:30:08.150657Z",
            "finished_at": "2020-04-24T10:40:19.555715Z",
            "error": {
                "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 167, in synchronize\n    dv.create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n    pb.done += task.result()  # download_count\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 81, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 184, in _run\n    async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n    self._resp = await self._coro\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 504, in _request\n    await resp.start(conn)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 847, in start\n    message, payload = await self._protocol.read()  # type: ignore  # noqa\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py\", line 591, in read\n    await self._waiter\n",
                "description": ""
            },
            "worker": "/pulp/api/v3/workers/e1b73364-ddd2-43d7-8874-b30ad4adff88/",
            "parent_task": null,
            "child_tasks": [],
            "task_group": null,
            "progress_reports": [
                {
                    "message": "Parsed Comps",
                    "code": "parsing.comps",
                    "state": "completed",
                    "total": 91,
                    "done": 91,
                    "suffix": null
                },
                {
                    "message": "Parsed Advisories",
                    "code": "parsing.advisories",
                    "state": "completed",
                    "total": 4325,
                    "done": 4325,
                    "suffix": null
                },
                {
                    "message": "Downloading Artifacts",
                    "code": "downloading.artifacts",
                    "state": "failed",
                    "total": null,
                    "done": 24696,
                    "suffix": null
                },
                {
                    "message": "Parsed Packages",
                    "code": "parsing.packages",
                    "state": "canceled",
                    "total": 28773,
                    "done": 27028,
                    "suffix": null
                },
                {
                    "message": "Downloading Metadata Files",
                    "code": "downloading.metadata",
                    "state": "canceled",
                    "total": null,
                    "done": 5,
                    "suffix": null
                },
                {
                    "message": "Associating Content",
                    "code": "associating.content",
                    "state": "canceled",
                    "total": null,
                    "done": 29032,
                    "suffix": null
                }
            ],
            "created_resources": [],
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/169ae56e-08a5-4fdd-ac71-e8624ca442bb/",
                "/pulp/api/v3/remotes/rpm/rpm/bf23c689-5e23-4286-91f0-48e6704083d4/"
            ]
        }

and

        {
            "pulp_href": "/pulp/api/v3/tasks/5d420b1d-dcf0-40a3-b564-1fa2b78b1cc6/",
            "pulp_created": "2020-04-24T10:45:59.167139Z",
            "state": "failed",
            "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
            "started_at": "2020-04-24T10:45:59.416371Z",
            "finished_at": "2020-04-24T10:57:29.180060Z",
            "error": {
                "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 167, in synchronize\n    dv.create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n    pb.done += task.result()  # download_count\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 81, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 184, in _run\n    async with self.session.get(self.url, proxy=self.proxy, auth=self.auth) as response:\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n    self._resp = await self._coro\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client.py\", line 504, in _request\n    await resp.start(conn)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 847, in start\n    message, payload = await self._protocol.read()  # type: ignore  # noqa\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/streams.py\", line 591, in read\n    await self._waiter\n",
                "description": ""
            },
            "worker": "/pulp/api/v3/workers/e1b73364-ddd2-43d7-8874-b30ad4adff88/",
            "parent_task": null,
            "child_tasks": [],
            "task_group": null,
            "progress_reports": [
                {
                    "message": "Parsed Comps",
                    "code": "parsing.comps",
                    "state": "completed",
                    "total": 91,
                    "done": 91,
                    "suffix": null
                },
                {
                    "message": "Parsed Advisories",
                    "code": "parsing.advisories",
                    "state": "completed",
                    "total": 4325,
                    "done": 4325,
                    "suffix": null
                },
                {
                    "message": "Downloading Artifacts",
                    "code": "downloading.artifacts",
                    "state": "failed",
                    "total": null,
                    "done": 475,
                    "suffix": null
                },
                {
                    "message": "Parsed Packages",
                    "code": "parsing.packages",
                    "state": "canceled",
                    "total": 28773,
                    "done": 28029,
                    "suffix": null
                },
                {
                    "message": "Downloading Metadata Files",
                    "code": "downloading.metadata",
                    "state": "canceled",
                    "total": null,
                    "done": 5,
                    "suffix": null
                },
                {
                    "message": "Associating Content",
                    "code": "associating.content",
                    "state": "canceled",
                    "total": null,
                    "done": 29500,
                    "suffix": null
                }
            ],
            "created_resources": [],
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/169ae56e-08a5-4fdd-ac71-e8624ca442bb/",
                "/pulp/api/v3/remotes/rpm/rpm/bf23c689-5e23-4286-91f0-48e6704083d4/"
            ]
        },

If you need me to gather more logs or if there is a way to get file names being failed upon, then please let me know

Thanks

Matt

Related issues

Related to Pulp - Task #6624: Document how Pulp handles the ServerDisconnectedError and whyCLOSED - DUPLICATE

Actions
Related to Pulp - Story #8881: As a user, Pulp will retry downloads by default 3 times and I can configure this value on my RemoteCLOSED - CURRENTRELEASEdalley

Actions
Actions
Send by e-mail Copy link

Also available in: Atom PDF