Task #3663

Updated by ttereshc about 6 years ago

If there are many orphans to be removed, the orphan cleanup task can take several hours. 

 The current orphan cleanup task performs everything without concurrency. 
 We can introduce a new orphan cleanup task that breaks up the list of orphans into groups and performs cleanup on them in parallel. 

 A good start is to "cprofile": "cprofile": this task to see where the bottleneck is and to determine if parallel execution will help. 

 Some facts to consider: 
  - orphan cleanup task is a task without resource reservation 
  - currently multiple orphan cleanup tasks can run at the same time (with the same list of orphans) and that increases mongo cpu usage and significantly slows down performance. See #3045. 

 The outcome of this task should be another task with detailed proposal on how to improve orphan cleanup performance. 
 As a compromise, if improving orphan cleanup task is not an option, consider providing an orphan cleanup script, which users can run in case they have a lot of orphans.