Project

Profile

Help

Issue #7212

Adjust download concurrency

Added by bmbouter 2 months ago. Updated about 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 79
Quarter:

Description

There are reports that the download concurrency is too high, like this one: https://pulp.plan.io/issues/7186#note-4

So we should lower the concurrency limit. We need to balance the stability gained from a lower concurrency against the lost speeds from parallel fetching.

Can someone measure the runtime of a policy=immediate sync with the following download_concurrency values:

20, 15, 10, 7, 5, 3

Pulp2 had a value of 5. Other download softwares like bandersnatch have a value of 3, so one of those is the most likely choice. We should understand how the performance degrades though so running this test would be nice.


Related issues

Related to RPM Support - Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failingCLOSED - DUPLICATE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 13594612 View on GitHub
Added by daviddavis about 2 months ago

Changing default download_concurrency to 10

fixes #7212

History

#1 Updated by bmbouter 2 months ago

  • Description updated (diff)

#2 Updated by bmbouter 2 months ago

  • Related to Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failing added

#3 Updated by ggainey 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
  • Sprint set to Sprint 78

Sync Centos7 3 times at each of the proposed concurrency settings, record wall-clock time here. Bring back to open floor when done and report.

#4 Updated by ggainey 2 months ago

Some concurrency test results written up here : https://hackmd.io/@ggainey/pulp3_sync_concurrency

tl;dr : 5 is half as slow as 20, 3 is half that. A setting of 5 or 7 looks like a reasonable compromise (imho)

#5 Updated by fao89 2 months ago

  • Triaged changed from No to Yes

#6 Updated by ggainey 2 months ago

Discussion in bug-triage, #pulp-dev and pulp-dev@ suggests:

  • move default to 10
  • perform a migration to move any currently at 20, to 10
  • document

#7 Updated by ggainey 2 months ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (ggainey)

#8 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 78 to Sprint 79

#9 Updated by daviddavis about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#10 Updated by pulpbot about 2 months ago

  • Status changed from ASSIGNED to POST

#11 Updated by daviddavis about 2 months ago

  • Status changed from POST to MODIFIED

#12 Updated by dkliban@redhat.com about 2 months ago

  • Sprint/Milestone set to 3.6.0

#13 Updated by pulpbot about 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF