Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #7212

closed

Adjust download concurrency

Added by bmbouter almost 4 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
daviddavis
Category:
-
Sprint/Milestone:
3.6.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 79
Quarter:

Description

There are reports that the download concurrency is too high, like this one: https://pulp.plan.io/issues/7186#note-4

So we should lower the concurrency limit. We need to balance the stability gained from a lower concurrency against the lost speeds from parallel fetching.

Can someone measure the runtime of a policy=immediate sync with the following download_concurrency values:

20, 15, 10, 7, 5, 3

Pulp2 had a value of 5. Other download softwares like bandersnatch have a value of 3, so one of those is the most likely choice. We should understand how the performance degrades though so running this test would be nice.

Related issues

Related to RPM Support - Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failingCLOSED - DUPLICATEActions
Related to Pulp - Issue #8878: default Download Concurrency is too high for many centos mirrorsCLOSED - DUPLICATEdalleyActions
Actions
Copy link
 #1

Updated by bmbouter almost 4 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by bmbouter almost 4 years ago

  • Related to Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failing added
Actions
Copy link
 #3

Updated by ggainey almost 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
  • Sprint set to Sprint 78

Sync Centos7 3 times at each of the proposed concurrency settings, record wall-clock time here. Bring back to open floor when done and report.

Actions
Copy link
 #4

Updated by ggainey over 3 years ago

Some concurrency test results written up here : https://hackmd.io/@ggainey/pulp3_sync_concurrency

tl;dr : 5 is half as slow as 20, 3 is half that. A setting of 5 or 7 looks like a reasonable compromise (imho)

Actions
Copy link
 #5

Updated by fao89 over 3 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #6

Updated by ggainey over 3 years ago

Discussion in bug-triage, #pulp-dev and pulp-dev@ suggests:

  • move default to 10
  • perform a migration to move any currently at 20, to 10
  • document
Actions
Copy link
 #7

Updated by ggainey over 3 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (ggainey)
Actions
Copy link
 #8

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 78 to Sprint 79
Actions
Copy link
 #9

Updated by daviddavis over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions
Copy link
 #10

Updated by pulpbot over 3 years ago

  • Status changed from ASSIGNED to POST

Added by daviddavis over 3 years ago

Revision 13594612 | View on GitHub

Changing default download_concurrency to 10

fixes #7212

Actions
Copy link
 #11

Updated by daviddavis over 3 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #12

Updated by dkliban@redhat.com over 3 years ago

  • Sprint/Milestone set to 3.6.0
Actions
Copy link
 #13

Updated by pulpbot over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Copy link
 #14

Updated by jsherril@redhat.com almost 3 years ago

  • Related to Issue #8878: default Download Concurrency is too high for many centos mirrors added
Actions
Send by e-mail Copy link

Also available in: Atom PDF