Task #3663: Investigate orphan cleanup performance - Pulp

Actions

Send by e-mail Copy link

Task #3663

closed

Investigate orphan cleanup performance

Added by ttereshc over 6 years ago. Updated over 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

jortel@redhat.com

Category:

Sprint/Milestone:

Start date:

Due date:

% Done:

100%

Estimated time:

Platform Release:

2.16.2

Groomed:

Yes

Sprint Candidate:

Yes

Tags:

Pulp 2

Sprint:

Sprint 38

Quarter:

Description

If there are many orphans to be removed, the orphan cleanup task can take several hours.

The current orphan cleanup task performs everything without concurrency.
We can introduce a new orphan cleanup task that breaks up the list of orphans into groups and performs cleanup on them in parallel.

A good start is to cprofile this task to see where the bottleneck is and to determine if parallel execution will help.

Some facts to consider:
- orphan cleanup task is a task without resource reservation
- currently multiple orphan cleanup tasks can run at the same time (with the same list of orphans) and that increases mongo cpu usage and significantly slows down performance. See #3045.

The outcome of this task should be another task with detailed proposal on how to improve orphan cleanup performance.
As a compromise, if improving orphan cleanup task is not an option, consider providing an orphan cleanup script, which users can run in case they have a lot of orphans.

Actions

Copy link

Updated by ttereshc over 6 years ago

Description updated (diff)

Actions

Copy link

Updated by ttereshc over 6 years ago

Tracker changed from Issue to Task
% Done set to 0

Actions

Copy link

Updated by dkliban@redhat.com over 6 years ago

Groomed changed from No to Yes
Sprint Candidate changed from No to Yes

Actions

Copy link

Updated by ttereshc over 6 years ago

Sprint set to Sprint 37

Actions

Copy link

Updated by jortel@redhat.com over 6 years ago

Status changed from NEW to ASSIGNED
Assignee set to jortel@redhat.com

Actions

Copy link

Updated by jortel@redhat.com over 6 years ago

Performance improved 62x by adding a missing index. This change also substantially reduces the mongod CPU usage. Support engineers can safely create the index (as done below) on customer sites to mitigate the problem until pulp code change is released.

root@oser501800 content]# pulp-admin orphan list
+----------------------------------------------------------------------+
                                Summary
+----------------------------------------------------------------------+

Distribution:            1
Docker Blob:             0
Docker Image:            0
Docker Manifest:         0
Docker Tag:              0
Drpm:                    18105
Erratum:                 158
Iso:                     0
Ostree:                  0
Package Category:        0
Package Environment:     0
Package Group:           196955
Package Langpacks:       0
Puppet Module:           0
Rpm:                     1521
Srpm:                    647
Yum Repo Metadata File:  334
Total:                   217721

[root@oser501800 mongodb]# time pulp-admin orphan remove --all
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

real    561m23.588s  (9 hours, 35 minutes)
user    4m8.252s
sys    0m10.966s

FIX

# mongo pulp_database
>
> db.lazy_content_catalog.createIndex({unit_id:1,unit_type_id:1})
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 3,
    "numIndexesAfter" : 4,
    "ok" : 1
}

RESTORE DB

root@oser501800 content]# pulp-admin orphan list
+----------------------------------------------------------------------+
                                Summary
+----------------------------------------------------------------------+

Distribution:            1
Docker Blob:             0
Docker Image:            0
Docker Manifest:         0
Docker Tag:              0
Drpm:                    18105
Erratum:                 158
Iso:                     0
Ostree:                  0
Package Category:        0
Package Environment:     0
Package Group:           196955
Package Langpacks:       0
Puppet Module:           0
Rpm:                     1521
Srpm:                    647
Yum Repo Metadata File:  334
Total:                   217721

[root@oser501800 mongodb]# time pulp-admin orphan remove --all
This command may be exited via ctrl+c without affecting the request.

[|]
Running...

Task Succeeded

real    9m16.914s
user    0m4.197s
sys    0m0.197s

Actions

Copy link