Issue #676
closed
Errata unit copy with recursive copy and dep solve causes worker to balloon memory usage and never free
Description
Description of problem:
When doing a unit copy of a single errata with a large source repo if recursive and resolve_dependencies are set to true the pulp worker doing the copy will balloon to a very large memory size and never free that memory even when the unit copy is completed.
post "https://localhost/pulp/api/v2/repositories/new_repo/actions/associate/"
{"source_repo_id":"bigrepo","criteria":{"type_ids":["erratum"],"filters":{"association":{"unit_id":{"$in":["26b996e9-ab64-4f09-ba94-7b846c71573b"]}}}},"override_config":{"recursive":true,"resolve_dependencies":true}}
The source repo in this case was rhel 6 x86_64, containing ~14400 packages and ~2800 errata.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
apache 28498 0.0 25.3 1843224 1483900 ? S Jan23 0:28 /usr/bin/python -m celery.__main__ worker -c 1 -n reserved_resource_worker-2@abed.usersys.redhat.com --events --app=
Version-Release number of selected component (if applicable):
2.5.1
How reproducible:
always
Steps to Reproduce:
1. Perform a recursive errata unit copy with dep resolution turned on
2. Watch memory usage for processes "watch 'ps aux | sort -nk +4 | tail'" will show the top processes by memory usage.
Actual results:
pulp worker will quickly creep up in memory usage and never go back down.
Expected results:
Memory usage goes back down after unit copy
Additional info:
+ This bug was cloned from Bugzilla Bug #1185868 +
Files
Updated by jsherril@redhat.com about 8 years ago
The errata in this example was: RHBA-2014:1965
+ This comment was cloned from Bugzilla #1185868 comment 1 +
Updated by cduryee about 8 years ago
repro steps:
create and sync a rhel6 repo
create a second empty rpm repo (copyrepo in this case)
run this:
pulp-admin rpm repo copy errata --from-repo-id el6 --to-repo-id copyrepo --str-eq="id=RHBA-2014:1965" --recursive
This will either OOM or hold onto a lot of memory.
+ This comment was cloned from Bugzilla #1185868 comment 2 +
Updated by cduryee about 8 years ago
I was not able to get this to repro with 4GB mem on 2.6.0. The repo copy (sans "fields" attribute) ate a lot of mem but gave it up at the end of the operation.
If you still have the 2.5.x system that exhibits this behavior, we can take a look at it tomorrow.
+ This comment was cloned from Bugzilla #1185868 comment 3 +
Updated by cduryee about 8 years ago
Justin,
I did further research on this yesterday and today. I set up the following scenarios, all with python 2.6:
- copy an erratum from RHEL 6.6 to 6.1
- copy an erratum for RHEL 5.11 to 5.6
Both of these make the memory usage grow, but it appears to be stable after growing. I assume this is from cPython not calling free() after performing these operations.
I'm going to mark this as CLOSED/NOTABUG since we already have a BZ to fix the underlying issue (https://bugzilla.redhat.com/show_bug.cgi?id=1158545). If the amount of memory in this particular scenario becomes problematic (OOMs, etc), let us know and we can try to come up with a workaround. FWIW I was not able to get it to OOM on a 4GB machine.
+ This comment was cloned from Bugzilla #1185868 comment 4 +
Updated by cduryee about 8 years ago
reopening bz
+ This comment was cloned from Bugzilla #1185868 comment 5 +
Updated by cduryee about 8 years ago
https://github.com/pulp/pulp/pull/1636
and
https://github.com/pulp/pulp_rpm/pull/650
+ This comment was cloned from Bugzilla #1185868 comment 6 +
Updated by cduryee about 8 years ago
merged to 2.6-dev and master
QE note to repro: comment #2 is incorrect, the first repo needs to be RHEL 6.6 and the second RHEL 6.1. After that, copying with --recursive should do the trick.
+ This comment was cloned from Bugzilla #1185868 comment 7 +
Updated by mhrivnak about 8 years ago
reproducer script
This is the script I was using to reproduce the problem. Note the comments at the top that tell you what repos you need to sync before running it.
If you use this on rhel6, you'll have to modify the "systemctl" statement accordingly.
+ This comment was cloned from Bugzilla #1185868 comment 8 +
Updated by pthomas@redhat.com about 8 years ago
- Status changed from 5 to 6
verified
Tested with the script provided and the memory consumption has reduced.
[root@cloud-qe-12 ~]# rpm -qa pulp-server
pulp-server-2.6.1-0.2.beta.el6.noarch
[root@cloud-qe-12 ~]#
Updated by dkliban@redhat.com almost 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
.