Investigate orphan cleanup performance
If there are many orphans to be removed, the orphan cleanup task can take several hours.
The current orphan cleanup task performs everything without concurrency.
We can introduce a new orphan cleanup task that breaks up the list of orphans into groups and performs cleanup on them in parallel.
A good start is to cprofile this task to see where the bottleneck is and to determine if parallel execution will help.
Some facts to consider:
- orphan cleanup task is a task without resource reservation
- currently multiple orphan cleanup tasks can run at the same time (with the same list of orphans) and that increases mongo cpu usage and significantly slows down performance. See #3045.
The outcome of this task should be another task with detailed proposal on how to improve orphan cleanup performance.
As a compromise, if improving orphan cleanup task is not an option, consider providing an orphan cleanup script, which users can run in case they have a lot of orphans.
Fix orphan delete performance by adding missing index. closes #3663