Task #3663
closed
Investigate orphan cleanup performance
100%
Description
If there are many orphans to be removed, the orphan cleanup task can take several hours.
The current orphan cleanup task performs everything without concurrency.
We can introduce a new orphan cleanup task that breaks up the list of orphans into groups and performs cleanup on them in parallel.
A good start is to cprofile this task to see where the bottleneck is and to determine if parallel execution will help.
Some facts to consider:
- orphan cleanup task is a task without resource reservation
- currently multiple orphan cleanup tasks can run at the same time (with the same list of orphans) and that increases mongo cpu usage and significantly slows down performance. See #3045.
The outcome of this task should be another task with detailed proposal on how to improve orphan cleanup performance.
As a compromise, if improving orphan cleanup task is not an option, consider providing an orphan cleanup script, which users can run in case they have a lot of orphans.
.
Fix orphan delete performance by adding missing index. closes #3663