Project

Profile

Help

Issue #1330

closed

celery worker memory leak and high CPU after restarting qpidd many times

Added by bmbouter over 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
2.6.4
Platform Release:
OS:
Fedora 22
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

To reproduce:

1. Start Pulp normally. I reproduced this on 2.6.4.
2. While Pulp is running, restart qpidd very often (every few seconds). I do that with this shell script:

while true
do sudo service qpidd restart
sleep 1.1
sleep $(($((RANDOM))%2))
done

3. Observe that over a few minutes the CPU will grow. You can watch this by periodically listing the processes and looking at the CPU column. I do this with

ps -awfux | grep celery

4. Once your processes are consuming lots of CPU for no apparent reason, stop the qpidd restart script and ensure that qpidd is started.
5. Observe that your CPU never goes back down even though Pulp is not doing any work and qpidd is running normally.
6. Despair

You can recover from this by restarting the celery processes. The original reporter who told me about this also indicated there was a memory leak, but I haven't confirmed this.


Related issues

Related to Pulp - Story #2632: As a developer I want to reevaluate worker issues to see if they have been resolved by moving from Celery3 to Celery4CLOSED - WONTFIX

Actions

Also available in: Atom PDF