Project

Profile

Help

Task #3663

Investigate orphan cleanup performance

Added by ttereshc over 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.16.2
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 38
Quarter:

Description

If there are many orphans to be removed, the orphan cleanup task can take several hours.

The current orphan cleanup task performs everything without concurrency.
We can introduce a new orphan cleanup task that breaks up the list of orphans into groups and performs cleanup on them in parallel.

A good start is to cprofile this task to see where the bottleneck is and to determine if parallel execution will help.

Some facts to consider:
- orphan cleanup task is a task without resource reservation
- currently multiple orphan cleanup tasks can run at the same time (with the same list of orphans) and that increases mongo cpu usage and significantly slows down performance. See #3045.

The outcome of this task should be another task with detailed proposal on how to improve orphan cleanup performance.
As a compromise, if improving orphan cleanup task is not an option, consider providing an orphan cleanup script, which users can run in case they have a lot of orphans.

Associated revisions

Revision eda385b1 View on GitHub
Added by jortel@redhat.com over 3 years ago

Fix orphan delete performance by adding missing index. closes #3663

Revision e098dc9c View on GitHub
Added by jortel@redhat.com over 3 years ago

Fix orphan delete performance by adding missing index. closes #3663

(cherry picked from commit eda385b14bcb988f7e0704d5fa911806b9a4bff0)

History

#1 Updated by ttereshc over 3 years ago

  • Description updated (diff)

#2 Updated by ttereshc over 3 years ago

  • Tracker changed from Issue to Task
  • % Done set to 0

#4 Updated by dkliban@redhat.com over 3 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#5 Updated by ttereshc over 3 years ago

  • Sprint set to Sprint 37

#6 Updated by jortel@redhat.com over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#7 Updated by jortel@redhat.com over 3 years ago

Performance improved 62x by adding a missing index. This change also substantially reduces the mongod CPU usage. Support engineers can safely create the index (as done below) on customer sites to mitigate the problem until pulp code change is released.

root@oser501800 content]# pulp-admin orphan list
+----------------------------------------------------------------------+
                                Summary
+----------------------------------------------------------------------+

Distribution:            1
Docker Blob:             0
Docker Image:            0
Docker Manifest:         0
Docker Tag:              0
Drpm:                    18105
Erratum:                 158
Iso:                     0
Ostree:                  0
Package Category:        0
Package Environment:     0
Package Group:           196955
Package Langpacks:       0
Puppet Module:           0
Rpm:                     1521
Srpm:                    647
Yum Repo Metadata File:  334
Total:                   217721

[root@oser501800 mongodb]# time pulp-admin orphan remove --all
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

real    561m23.588s  (9 hours, 35 minutes)
user    4m8.252s
sys    0m10.966s

FIX

# mongo pulp_database
>
> db.lazy_content_catalog.createIndex({unit_id:1,unit_type_id:1})
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 3,
    "numIndexesAfter" : 4,
    "ok" : 1
}

RESTORE DB

root@oser501800 content]# pulp-admin orphan list
+----------------------------------------------------------------------+
                                Summary
+----------------------------------------------------------------------+

Distribution:            1
Docker Blob:             0
Docker Image:            0
Docker Manifest:         0
Docker Tag:              0
Drpm:                    18105
Erratum:                 158
Iso:                     0
Ostree:                  0
Package Category:        0
Package Environment:     0
Package Group:           196955
Package Langpacks:       0
Puppet Module:           0
Rpm:                     1521
Srpm:                    647
Yum Repo Metadata File:  334
Total:                   217721
[root@oser501800 mongodb]# time pulp-admin orphan remove --all
This command may be exited via ctrl+c without affecting the request.

[|]
Running...

Task Succeeded

real    9m16.914s
user    0m4.197s
sys    0m0.197s

#8 Updated by jortel@redhat.com over 3 years ago

  • Status changed from ASSIGNED to POST

#9 Updated by rchan over 3 years ago

  • Sprint changed from Sprint 37 to Sprint 38

#10 Updated by jortel@redhat.com over 3 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#11 Updated by dkliban@redhat.com over 3 years ago

  • Platform Release set to 2.16.2

#12 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#13 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF