Project

Profile

Help

Issue #8878

closed

default Download Concurrency is too high for many centos mirrors

Added by jsherril@redhat.com over 2 years ago. Updated over 2 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 98
Quarter:

Description

The default download policy of 10 is still too high for many centos mirrors (and possibly the redhat cdn too).

Users have been reporting this error:

traceback: |2
        File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
          rv = job.perform()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
          self._result = self._execute()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 271, in synchronize
          dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 147, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 171, in run
          pb.done += task.result()  # download_count
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 197, in _handle_content_unit
          await asyncio.gather(*downloaders_for_content)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
          download_result = await downloader.run(extra_data=self.extra_data)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/base.py", line 227, in run
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 90, in _run
          to_return = await self._handle_response(response)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 185, in _handle_response
          chunk = await response.content.read(1048576)  # 1 megabyte
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read
          await self._wait("read")
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
          await waiter
    description: Response payload is not completed

We've had ~6 community posts about this:

https://community.theforeman.org/search?q=response%20payload%20is%20not%20complete

Typically we have them reduce the download concurrency and that seems to solve it. Stephen recently reduced it to 2 and still had a single failing sync (no other syncs going on at the same time)


Related issues

Related to RPM Support - Issue #6945: Sync error: Response payload is not completedCLOSED - WORKSFORMEActions
Related to RPM Support - Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failingCLOSED - DUPLICATEActions
Related to Pulp - Issue #7212: Adjust download concurrencyCLOSED - CURRENTRELEASEdaviddavisActions
Is duplicate of Pulp - Story #8881: As a user, Pulp will retry downloads by default 3 times and I can configure this value on my RemoteCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by jsherril@redhat.com over 2 years ago

  • Related to Issue #6945: Sync error: Response payload is not completed added
Actions #2

Updated by jsherril@redhat.com over 2 years ago

  • Related to Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failing added
Actions #3

Updated by jsherril@redhat.com over 2 years ago

  • Related to Issue #7212: Adjust download concurrency added
Actions #4

Updated by dalley over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #5

Updated by dalley over 2 years ago

  • Status changed from ASSIGNED to POST
Actions #6

Updated by ipanova@redhat.com over 2 years ago

  • Sprint set to Sprint 98
Actions #7

Updated by dkliban@redhat.com over 2 years ago

  • Triaged changed from No to Yes
Actions #8

Updated by jsherril@redhat.com over 2 years ago

After some further testing, i do not believe this is concurrency related at all, instead is likely hiccups in the network.

Retrying the download is likely to solve the issue.

Actions #9

Updated by bmbouter over 2 years ago

  • Status changed from POST to CLOSED - DUPLICATE

Closing as a duplicate of https://pulp.plan.io/issues/8881 per recommendation from @jsherrill

Actions #10

Updated by bmbouter over 2 years ago

  • Is duplicate of Story #8881: As a user, Pulp will retry downloads by default 3 times and I can configure this value on my Remote added

Also available in: Atom PDF