Project

Profile

Help

Issue #7212

closed

Adjust download concurrency

Added by bmbouter about 2 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 79
Quarter:

Description

There are reports that the download concurrency is too high, like this one: https://pulp.plan.io/issues/7186#note-4

So we should lower the concurrency limit. We need to balance the stability gained from a lower concurrency against the lost speeds from parallel fetching.

Can someone measure the runtime of a policy=immediate sync with the following download_concurrency values:

20, 15, 10, 7, 5, 3

Pulp2 had a value of 5. Other download softwares like bandersnatch have a value of 3, so one of those is the most likely choice. We should understand how the performance degrades though so running this test would be nice.


Related issues

Related to RPM Support - Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failingCLOSED - DUPLICATEActions
Related to Pulp - Issue #8878: default Download Concurrency is too high for many centos mirrorsCLOSED - DUPLICATEdalleyActions
Actions #1

Updated by bmbouter about 2 years ago

  • Description updated (diff)
Actions #2

Updated by bmbouter about 2 years ago

  • Related to Issue #7186: Immediate syncing of some CentOS 6 and 7 repos failing added
Actions #3

Updated by ggainey about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
  • Sprint set to Sprint 78

Sync Centos7 3 times at each of the proposed concurrency settings, record wall-clock time here. Bring back to open floor when done and report.

Actions #4

Updated by ggainey about 2 years ago

Some concurrency test results written up here : https://hackmd.io/@ggainey/pulp3_sync_concurrency

tl;dr : 5 is half as slow as 20, 3 is half that. A setting of 5 or 7 looks like a reasonable compromise (imho)

Actions #5

Updated by fao89 about 2 years ago

  • Triaged changed from No to Yes
Actions #6

Updated by ggainey about 2 years ago

Discussion in bug-triage, #pulp-dev and pulp-dev@ suggests:

  • move default to 10
  • perform a migration to move any currently at 20, to 10
  • document
Actions #7

Updated by ggainey about 2 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (ggainey)
Actions #8

Updated by rchan about 2 years ago

  • Sprint changed from Sprint 78 to Sprint 79
Actions #9

Updated by daviddavis about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions #10

Updated by pulpbot about 2 years ago

  • Status changed from ASSIGNED to POST

Added by daviddavis about 2 years ago

Revision 13594612

Changing default download_concurrency to 10

fixes #7212

Actions #11

Updated by daviddavis about 2 years ago

  • Status changed from POST to MODIFIED
Actions #12

Updated by dkliban@redhat.com about 2 years ago

  • Sprint/Milestone set to 3.6.0
Actions #13

Updated by pulpbot about 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #14

Updated by jsherril@redhat.com over 1 year ago

  • Related to Issue #8878: default Download Concurrency is too high for many centos mirrors added

Also available in: Atom PDF