Project

Profile

Help

Refactor #1013

Reduce sync time spent processing metadata up-front

Added by mhrivnak over 6 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
High
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Performance, Pulp 2
Sprint:
Quarter:

Description

The rpm sync workflow spends a lot of time early on parsing metadata (filelist.xml and other.xml) and writing it to disk in an indexed data structure. Actual downloading of files doesn't happen until afterward. This can take multiple minutes.

There are several options for improving this workflow.

  • start downloading before indexing is complete. This would require a new queue of completed downloads, and an additional thread that finishes processing and saving them once the indexing is complete.
  • Only index the rpms that will actually be downloaded. This would not help on the first sync (except in rare cases).
  • use the sqlite file (if available) to get this metadata instead of indexing the data ourselves. The downside here is we'd have to convert it to XML.
  • there may be other options

This refactor is to employ whatever improvements are reasonable to reduce the amount of time spent chewing on metadata before downloading starts.


Related issues

Related to RPM Support - Story #2567: As a user, I can quickly propogate one package change from one Pulp to another PulpCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Blocked by RPM Support - Refactor #874: Convert pulp_rpm to use MongoEngine ModelsCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by mhrivnak about 6 years ago

  • Blocked by Refactor #874: Convert pulp_rpm to use MongoEngine Models added

#2 Updated by mhrivnak almost 6 years ago

  • Platform Release set to 2.8.0

#3 Updated by jortel@redhat.com over 5 years ago

  • Priority changed from Normal to High
  • Platform Release deleted (2.8.0)

#4 Updated by mhrivnak over 5 years ago

  • Sprint Candidate changed from Yes to No

#6 Updated by dkliban@redhat.com over 4 years ago

  • Related to Story #2567: As a user, I can quickly propogate one package change from one Pulp to another Pulp added

#7 Updated by dkliban@redhat.com over 4 years ago

  • Tags Sync Performance added

#8 Updated by dkliban@redhat.com over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

This is going to be addressed by the stream processing design of Pulp 3.

#9 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

#10 Updated by bmbouter over 1 year ago

  • Tags Performance added
  • Tags deleted (Sync Performance)

Please register to edit this issue

Also available in: Atom PDF