Project

Profile

Help

Story #4527

closed

Improve performance of rpm duplicate nevra check

Added by rmcgover over 5 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.19.0
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 50
Quarter:

Description

In current versions of Pulp 2.x, uploading an RPM to a repo will remove other RPMs with the same NEVRA.

Currently, we are upgrading from an old version of Pulp 2.7, and I've found that performance of import_uploaded_unit tasks for RPMs has regressed significantly. In Pulp 2.7, imports would usually take around 0.5s. In Pulp 2-master, imports to the same repos have taken from 8 to 130 seconds, depending on the size of the repo.

By debugging I've found most of the time is spent in this duplicate check (remove_unit_duplicate_nevra).

This issue is for improving the performance of remove_unit_duplicate_nevra to reduce the severity of the performance regression.


Related issues

Copied to RPM Support - Test #4566: Improve performance of rpm duplicate nevra checkCLOSED - COMPLETEbherringActions

Added by rmcgover over 5 years ago

Revision 3bfdbd84 | View on GitHub

Improve performance of remove_unit_duplicate_nevra

This function, which is used whenever a new RPM is uploaded, was slower than necessary.

The old implementation would first use find_repo_content_units, which queries for all unit IDs of the required type in the repo and then performs a unit query combining the relevant NEVRA with the IDs.

In fact finding all those unit IDs is measurably slow for a large repo, and it's much faster (while still correct) to simply search for the RPMs to remove directly. It's harmless if this finds some RPMs with same NEVRA which are already not in the repo.

On our installation, for a repo with ~24000 RPMs, this reduced the runtime of this method from ~8 seconds to <0.1 seconds.

fixes #4527 https://pulp.plan.io/issues/4527

Actions #2

Updated by rmcgover over 5 years ago

  • Status changed from ASSIGNED to POST
Actions #3

Updated by ttereshc over 5 years ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 50
Actions #4

Updated by rmcgover over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #5

Updated by ttereshc over 5 years ago

  • Platform Release set to 2.19.0
Actions #6

Updated by ttereshc over 5 years ago

  • Sprint/Milestone set to 2.19.0
Actions #8

Updated by ttereshc over 5 years ago

  • Status changed from MODIFIED to 5
Actions #9

Updated by bherring over 5 years ago

  • Copied to Test #4566: Improve performance of rpm duplicate nevra check added
Actions #10

Updated by ttereshc over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #11

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF