Project

Profile

Help

Story #4708

Improve performance of publishing files with iso_distributor via the way of fast-forward

Added by Zhiming 9 months ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
2.20.0
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 53

Description

Improve performance of publishing with iso_distributor via the way of fast-forward

This function, which is used to publishing a repo with iso_distributor, takes around 35 minutes (there are ~88k units in the repository)

The old implementation would generate symlinks for all units in the repository then copy to target published directories. If there are huge mount of units in the repository, the publish would be pretty slow due to those file operations.

Actually, just a few files are added/changed before every publish. So if just generate symlinks for those incremental or changed files, the publish will get faster.

On our installation, for a repo with 71K, this reduced the runtime of this publish from ~23 minutes to ~30 seconds.


Related issues

Related to RPM Support - Issue #4857: Pulp 2 Nightly Regression in test_iso_crud.ISOUpdateTestCase.test_all CLOSED - DUPLICATE Actions

Associated revisions

Revision f6685faf View on GitHub
Added by Zhiming 9 months ago

Improve performance of publish with iso_distributor via fast-forward

Only symlinks for those incremental or changed files are generated
and copied to target directories, then the publish gets faster.

ref #4708
https://pulp.plan.io/issues/4708

update unit test because os.makedirs is now called conditionally

add missing parameters

History

#3 Updated by ttereshc 9 months ago

  • Status changed from NEW to POST

#4 Updated by amacdona@redhat.com 9 months ago

  • Triaged changed from No to Yes

#5 Updated by ipanova@redhat.com 8 months ago

  • Status changed from POST to MODIFIED

#6 Updated by ttereshc 8 months ago

  • Blocks Release deleted (2.17.z)

#7 Updated by ttereshc 8 months ago

  • Tracker changed from Issue to Story
  • Assignee set to Zhiming
  • % Done set to 0
  • Groomed changed from No to Yes
  • Sprint set to Sprint 53

#8 Updated by ttereshc 8 months ago

  • Related to Issue #4857: Pulp 2 Nightly Regression in test_iso_crud.ISOUpdateTestCase.test_all added

#9 Updated by ttereshc 8 months ago

  • Platform Release set to 2.20.0

#10 Updated by ttereshc 7 months ago

  • Sprint/Milestone set to 2.20.0

#11 Updated by ttereshc 7 months ago

  • Status changed from MODIFIED to ON_QA

#12 Updated by ttereshc 6 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF