Errata unit copy with recursive copy and dep solve causes worker to balloon memory usage and never free
Description of problem:
When doing a unit copy of a single errata with a large source repo if recursive and resolve_dependencies are set to true the pulp worker doing the copy will balloon to a very large memory size and never free that memory even when the unit copy is completed.
The source repo in this case was rhel 6 x86_64, containing ~14400 packages and ~2800 errata.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
apache 28498 0.0 25.3 1843224 1483900 ? S Jan23 0:28 /usr/bin/python -m celery.__main__ worker -c 1 -n email@example.com --events --app=
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Perform a recursive errata unit copy with dep resolution turned on
2. Watch memory usage for processes "watch 'ps aux | sort -nk +4 | tail'" will show the top processes by memory usage.
pulp worker will quickly creep up in memory usage and never go back down.
Memory usage goes back down after unit copy
+ This bug was cloned from Bugzilla Bug #1185868 +
#2 Updated by cduryee over 6 years ago
create and sync a rhel6 repo
create a second empty rpm repo (copyrepo in this case)
pulp-admin rpm repo copy errata --from-repo-id el6 --to-repo-id copyrepo --str-eq="id=RHBA-2014:1965" --recursive
This will either OOM or hold onto a lot of memory.
+ This comment was cloned from Bugzilla #1185868 comment 2 +
#3 Updated by cduryee over 6 years ago
I was not able to get this to repro with 4GB mem on 2.6.0. The repo copy (sans "fields" attribute) ate a lot of mem but gave it up at the end of the operation.
If you still have the 2.5.x system that exhibits this behavior, we can take a look at it tomorrow.
+ This comment was cloned from Bugzilla #1185868 comment 3 +
#4 Updated by cduryee over 6 years ago
I did further research on this yesterday and today. I set up the following scenarios, all with python 2.6:
- copy an erratum from RHEL 6.6 to 6.1
- copy an erratum for RHEL 5.11 to 5.6
Both of these make the memory usage grow, but it appears to be stable after growing. I assume this is from cPython not calling free() after performing these operations.
I'm going to mark this as CLOSED/NOTABUG since we already have a BZ to fix the underlying issue (https://bugzilla.redhat.com/show_bug.cgi?id=1158545). If the amount of memory in this particular scenario becomes problematic (OOMs, etc), let us know and we can try to come up with a workaround. FWIW I was not able to get it to OOM on a 4GB machine.
+ This comment was cloned from Bugzilla #1185868 comment 4 +
#8 Updated by mhrivnak over 6 years ago
This is the script I was using to reproduce the problem. Note the comments at the top that tell you what repos you need to sync before running it.
If you use this on rhel6, you'll have to modify the "systemctl" statement accordingly.
+ This comment was cloned from Bugzilla #1185868 comment 8 +
Please register to edit this issue