Project

Profile

Help

Task #3663

closed

Investigate orphan cleanup performance

Added by ttereshc almost 6 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.16.2
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 38
Quarter:

Description

If there are many orphans to be removed, the orphan cleanup task can take several hours.

The current orphan cleanup task performs everything without concurrency.
We can introduce a new orphan cleanup task that breaks up the list of orphans into groups and performs cleanup on them in parallel.

A good start is to cprofile this task to see where the bottleneck is and to determine if parallel execution will help.

Some facts to consider:
- orphan cleanup task is a task without resource reservation
- currently multiple orphan cleanup tasks can run at the same time (with the same list of orphans) and that increases mongo cpu usage and significantly slows down performance. See #3045.

The outcome of this task should be another task with detailed proposal on how to improve orphan cleanup performance.
As a compromise, if improving orphan cleanup task is not an option, consider providing an orphan cleanup script, which users can run in case they have a lot of orphans.

Actions #1

Updated by ttereshc almost 6 years ago

  • Description updated (diff)
Actions #2

Updated by ttereshc almost 6 years ago

  • Tracker changed from Issue to Task
  • % Done set to 0
Actions #4

Updated by dkliban@redhat.com almost 6 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #5

Updated by ttereshc almost 6 years ago

  • Sprint set to Sprint 37
Actions #6

Updated by jortel@redhat.com almost 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
Actions #7

Updated by jortel@redhat.com almost 6 years ago

Performance improved 62x by adding a missing index. This change also substantially reduces the mongod CPU usage. Support engineers can safely create the index (as done below) on customer sites to mitigate the problem until pulp code change is released.

root@oser501800 content]# pulp-admin orphan list
+----------------------------------------------------------------------+
                                Summary
+----------------------------------------------------------------------+

Distribution:            1
Docker Blob:             0
Docker Image:            0
Docker Manifest:         0
Docker Tag:              0
Drpm:                    18105
Erratum:                 158
Iso:                     0
Ostree:                  0
Package Category:        0
Package Environment:     0
Package Group:           196955
Package Langpacks:       0
Puppet Module:           0
Rpm:                     1521
Srpm:                    647
Yum Repo Metadata File:  334
Total:                   217721

[root@oser501800 mongodb]# time pulp-admin orphan remove --all
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

real    561m23.588s  (9 hours, 35 minutes)
user    4m8.252s
sys    0m10.966s

FIX

# mongo pulp_database
>
> db.lazy_content_catalog.createIndex({unit_id:1,unit_type_id:1})
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 3,
    "numIndexesAfter" : 4,
    "ok" : 1
}

RESTORE DB

root@oser501800 content]# pulp-admin orphan list
+----------------------------------------------------------------------+
                                Summary
+----------------------------------------------------------------------+

Distribution:            1
Docker Blob:             0
Docker Image:            0
Docker Manifest:         0
Docker Tag:              0
Drpm:                    18105
Erratum:                 158
Iso:                     0
Ostree:                  0
Package Category:        0
Package Environment:     0
Package Group:           196955
Package Langpacks:       0
Puppet Module:           0
Rpm:                     1521
Srpm:                    647
Yum Repo Metadata File:  334
Total:                   217721
[root@oser501800 mongodb]# time pulp-admin orphan remove --all
This command may be exited via ctrl+c without affecting the request.

[|]
Running...

Task Succeeded

real    9m16.914s
user    0m4.197s
sys    0m0.197s
Actions #8

Updated by jortel@redhat.com almost 6 years ago

  • Status changed from ASSIGNED to POST
Actions #9

Updated by rchan almost 6 years ago

  • Sprint changed from Sprint 37 to Sprint 38

Added by jortel@redhat.com almost 6 years ago

Revision eda385b1 | View on GitHub

Fix orphan delete performance by adding missing index. closes #3663

Actions #10

Updated by jortel@redhat.com almost 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #11

Updated by dkliban@redhat.com almost 6 years ago

  • Platform Release set to 2.16.2

Added by jortel@redhat.com over 5 years ago

Revision e098dc9c | View on GitHub

Fix orphan delete performance by adding missing index. closes #3663

(cherry picked from commit eda385b14bcb988f7e0704d5fa911806b9a4bff0)

Actions #12

Updated by ipanova@redhat.com over 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #13

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF