Issue #2362
closed
applicability calculation wastes time scanning a list
Description
Applicability calculation for errata spends most of its time scanning a large list of dicts to see if it contains a particular dict. Profiling shows that a normal applicability task spends about 95% of its time scanning this list. This is causing calculation to take much longer than is reasonably expected.
Considering a RHEL 6 repo with approximately 18000 rpms and 3700 errata, pulp does this:
make a list of dicts, each representing one rpm
for each errata:
for each rpm listed in the errata:
check if the rpm is in the list of dicts
The query happens here:
Profiling with cProfile showed that out of 81 seconds spent calculating applicability (both rpm and errata) for one consumer, 77 of those seconds were spent on this activity. The list was scanned more than 80000 times.
It should instead use a better data structure, like a set, that is optimized to perform membership queries in constant time.
.
errata applicability no longer spends most of its time scanning a list
https://pulp.plan.io/issues/2362 fixes #2362