Issue #1779
closed
Repo Copy Memory Usage
Description
I've been having some problems with memory usage when trying to set up my pulp server. It appears that the memory is bloating when copying from one repo to another. Below is some snipped output from top after tasks have finished and the system is idle.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3516 apache 20 0 3945872 3.141g 2848 S 0.0 55.4 77:13.18 celery
3513 apache 20 0 753988 139396 2076 S 0.0 2.3 1:20.90 celery
3063 apache 20 0 1156884 114524 3644 S 0.3 1.9 0:52.89 httpd
You can see that celery never let go of all of its memory. I'm running Pulp 2.8 on a single vm running CentOS Linux release 7.2.1511 (Core) , 2 CPUs, and 6GB RAM. Some times the memory usage becomes so constrained that the OOM killer will kill MongoDB or other processes.
The script that I run to set up my local pulp server is located here: https://gist.github.com/sysadmind/c9ceba7399a6695f8ca6
Related issues
Updated by sysadmind over 8 years ago
I believe this to be directly related to the 'pulp-admin rpm repo copy all' command that I'm running as mhrivnak had mentioned in IRC. Per his suggestion, I converted that command to several commands, each specifying a type of content to copy between the repos, and was able to complete the script with a max RAM usage of ~2GB. I believe that ~1GB of that usage (which is still left even after all of the tasks have finished) is directly related to celery. Below is the top output after running the script with mhrivnak's suggested changes.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3262 apache 20 0 1240244 593036 3600 S 0.0 10.0 145:13.20 celery
2933 mongodb 20 0 12.544g 223620 115664 S 0.0 3.8 14:32.96 mongod
3261 apache 20 0 774876 165176 2104 S 0.0 2.8 3:19.60 celery
Updated by bmbouter over 8 years ago
This issue should track the memory improvements of the copy all which it does well.
@sysadmind would you be interested in opening a separate issue to track the "celery does not free memory after work is completed issue?" Posting some pre and post task process-memory usage information would be great along with pulp-admin commands that you used in between the pre and post run. That way another user/dev could try to compare their results.
Updated by sysadmind over 8 years ago
I can certainly appreciate having a separate issue for the memory freeing issue that I believe I am having. If I can get some solid details together, I'll do just that. Thanks
Updated by mhrivnak over 8 years ago
- Triaged changed from No to Yes
This is a unfortunate behavior that's been in place for a long time, and we do have plans to fix it. Thanks for the report, and I'm glad the workaround is working for you.
Updated by haxus over 8 years ago
Hello I am affected as well on a CentOS 6.7 System with Pulp 2.8.2. In my case the worker took 8GB RAM and about 20GB Swap-Space. At the moment I an trying to run the Workaround, reporting later if this was working or not.
Updated by bizhang over 7 years ago
- Related to Story #2632: As a developer I want to reevaluate worker issues to see if they have been resolved by moving from Celery3 to Celery4 added
Updated by bmbouter over 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter over 5 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
.