Project

Profile

Help

Issue #676

Errata unit copy with recursive copy and dep solve causes worker to balloon memory usage and never free

Added by jsherril@redhat.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.5
Platform Release:
2.6.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem:

When doing a unit copy of a single errata with a large source repo if recursive and resolve_dependencies are set to true the pulp worker doing the copy will balloon to a very large memory size and never free that memory even when the unit copy is completed.

post "https://localhost/pulp/api/v2/repositories/new_repo/actions/associate/"

{"source_repo_id":"bigrepo","criteria":{"type_ids":["erratum"],"filters":{"association":{"unit_id":{"$in":["26b996e9-ab64-4f09-ba94-7b846c71573b"]}}}},"override_config":{"recursive":true,"resolve_dependencies":true}}

The source repo in this case was rhel 6 x86_64, containing ~14400 packages and ~2800 errata.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
apache 28498 0.0 25.3 1843224 1483900 ? S Jan23 0:28 /usr/bin/python -m celery.__main__ worker -c 1 -n --events --app=

Version-Release number of selected component (if applicable):
2.5.1

How reproducible:
always

Steps to Reproduce:
1. Perform a recursive errata unit copy with dep resolution turned on
2. Watch memory usage for processes "watch 'ps aux | sort -nk +4 | tail'" will show the top processes by memory usage.

Actual results:
pulp worker will quickly creep up in memory usage and never go back down.

Expected results:
Memory usage goes back down after unit copy

Additional info:

+ This bug was cloned from Bugzilla Bug #1185868 +

f37340f641ff643e2988e3d55909c5aa (570 Bytes) f37340f641ff643e2988e3d55909c5aa mhrivnak, 03/01/2015 12:21 AM

History

#1 Updated by jsherril@redhat.com over 6 years ago

The errata in this example was: RHBA-2014:1965

+ This comment was cloned from Bugzilla #1185868 comment 1 +

#2 Updated by cduryee over 6 years ago

repro steps:

create and sync a rhel6 repo

create a second empty rpm repo (copyrepo in this case)

run this:

pulp-admin rpm repo copy errata --from-repo-id el6 --to-repo-id copyrepo --str-eq="id=RHBA-2014:1965" --recursive

This will either OOM or hold onto a lot of memory.

+ This comment was cloned from Bugzilla #1185868 comment 2 +

#3 Updated by cduryee over 6 years ago

I was not able to get this to repro with 4GB mem on 2.6.0. The repo copy (sans "fields" attribute) ate a lot of mem but gave it up at the end of the operation.

If you still have the 2.5.x system that exhibits this behavior, we can take a look at it tomorrow.

+ This comment was cloned from Bugzilla #1185868 comment 3 +

#4 Updated by cduryee over 6 years ago

Justin,

I did further research on this yesterday and today. I set up the following scenarios, all with python 2.6:

  • copy an erratum from RHEL 6.6 to 6.1
  • copy an erratum for RHEL 5.11 to 5.6

Both of these make the memory usage grow, but it appears to be stable after growing. I assume this is from cPython not calling free() after performing these operations.

I'm going to mark this as CLOSED/NOTABUG since we already have a BZ to fix the underlying issue (https://bugzilla.redhat.com/show_bug.cgi?id=1158545). If the amount of memory in this particular scenario becomes problematic (OOMs, etc), let us know and we can try to come up with a workaround. FWIW I was not able to get it to OOM on a 4GB machine.

+ This comment was cloned from Bugzilla #1185868 comment 4 +

#5 Updated by cduryee over 6 years ago

reopening bz

+ This comment was cloned from Bugzilla #1185868 comment 5 +

#7 Updated by cduryee over 6 years ago

merged to 2.6-dev and master

QE note to repro: comment #2 is incorrect, the first repo needs to be RHEL 6.6 and the second RHEL 6.1. After that, copying with --recursive should do the trick.

+ This comment was cloned from Bugzilla #1185868 comment 7 +

#8 Updated by mhrivnak over 6 years ago

reproducer script

This is the script I was using to reproduce the problem. Note the comments at the top that tell you what repos you need to sync before running it.

If you use this on rhel6, you'll have to modify the "systemctl" statement accordingly.

+ This comment was cloned from Bugzilla #1185868 comment 8 +

#9 Updated by bmbouter over 6 years ago

  • Severity changed from High to 3. High

#10 Updated by bcourt over 6 years ago

  • Status changed from MODIFIED to 5

#12 Updated by pthomas@redhat.com over 6 years ago

  • Status changed from 5 to 6

verified
Tested with the script provided and the memory consumption has reduced.

[root@cloud-qe-12 ~]# rpm -qa pulp-server
pulp-server-2.6.1-0.2.beta.el6.noarch
[root@cloud-qe-12 ~]#

#13 Updated by bmbouter over 6 years ago

  • Tags deleted (Reopened)

#14 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#16 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF