Project

Profile

Help

Issue #8878

default Download Concurrency is too high for many centos mirrors

Added by jsherril@redhat.com 7 days ago. Updated 1 day ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 98
Quarter:

Description

The default download policy of 10 is still too high for many centos mirrors (and possibly the redhat cdn too).

Users have been reporting this error:

traceback: |2
        File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
          rv = job.perform()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
          self._result = self._execute()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 271, in synchronize
          dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 147, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 171, in run
          pb.done += task.result()  # download_count
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 197, in _handle_content_unit
          await asyncio.gather(*downloaders_for_content)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
          download_result = await downloader.run(extra_data=self.extra_data)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/base.py", line 227, in run
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 90, in _run
          to_return = await self._handle_response(response)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 185, in _handle_response
          chunk = await response.content.read(1048576)  # 1 megabyte
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read
          await self._wait("read")
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
          await waiter
    description: Response payload is not completed

We've had ~6 community posts about this:

https://community.theforeman.org/search?q=response%20payload%20is%20not%20complete

Typically we have them reduce the download concurrency and that seems to solve it. Stephen recently reduced it to 2 and still had a single failing sync (no other syncs going on at the same time)


Related issues

Related to RPM Support - Issue #6945: Sync error: Response payload is not completedCLOSED - WORKSFORME<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failingCLOSED - DUPLICATE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Issue #7212: Adjust download concurrencyCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Is duplicate of Pulp - Story #8881: As a user, Pulp will retry downloads by default 5 times and I can configure this value on my RemotePOST

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by jsherril@redhat.com 7 days ago

  • Related to Issue #6945: Sync error: Response payload is not completed added

#2 Updated by jsherril@redhat.com 7 days ago

  • Related to Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failing added

#3 Updated by jsherril@redhat.com 7 days ago

  • Related to Issue #7212: Adjust download concurrency added

#4 Updated by dalley 4 days ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#5 Updated by dalley 3 days ago

  • Status changed from ASSIGNED to POST

#6 Updated by ipanova@redhat.com 3 days ago

  • Sprint set to Sprint 98

#7 Updated by dkliban@redhat.com 1 day ago

  • Triaged changed from No to Yes

#8 Updated by jsherril@redhat.com 1 day ago

After some further testing, i do not believe this is concurrency related at all, instead is likely hiccups in the network.

Retrying the download is likely to solve the issue.

#9 Updated by bmbouter 1 day ago

  • Status changed from POST to CLOSED - DUPLICATE

Closing as a duplicate of https://pulp.plan.io/issues/8881 per recommendation from @jsherrill

#10 Updated by bmbouter 1 day ago

  • Is duplicate of Story #8881: As a user, Pulp will retry downloads by default 5 times and I can configure this value on my Remote added

Please register to edit this issue

Also available in: Atom PDF