Issue #1330
closed
celery worker memory leak and high CPU after restarting qpidd many times
Description
To reproduce:
1. Start Pulp normally. I reproduced this on 2.6.4.
2. While Pulp is running, restart qpidd very often (every few seconds). I do that with this shell script:
while true
do sudo service qpidd restart
sleep 1.1
sleep $(($((RANDOM))%2))
done
3. Observe that over a few minutes the CPU will grow. You can watch this by periodically listing the processes and looking at the CPU column. I do this with
ps -awfux | grep celery
4. Once your processes are consuming lots of CPU for no apparent reason, stop the qpidd restart script and ensure that qpidd is started.
5. Observe that your CPU never goes back down even though Pulp is not doing any work and qpidd is running normally.
6. Despair
You can recover from this by restarting the celery processes. The original reporter who told me about this also indicated there was a memory leak, but I haven't confirmed this.
- Triaged changed from No to Yes
I am not actively working on this so I'm setting back to NEW.
- Status changed from ASSIGNED to NEW
- Assignee deleted (
bmbouter)
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
- Sprint/Milestone set to 31
- Status changed from NEW to ASSIGNED
- Assignee set to bmbouter
- Sprint/Milestone changed from 31 to 32
- Sprint/Milestone changed from 32 to 33
- Sprint/Milestone changed from 33 to 34
- Related to Story #2632: As a developer I want to reevaluate worker issues to see if they have been resolved by moving from Celery3 to Celery4 added
In sprint planning we decided to wait on this issue, and let the upgrade to celery 4 fix it for us. We will try to reproduce on the new celery 4 stack, but expect it to be resolved.
See story #2632
- Status changed from ASSIGNED to NEW
- Assignee deleted (
bmbouter)
- Sprint/Milestone deleted (
34)
- Sprint Candidate deleted (
Yes)
- Status changed from NEW to CLOSED - WONTFIX
- Sprint Candidate set to No
This is not going to be fixed for Pulp 2. Pulp 3 does not rely on celery or qpid.
Also available in: Atom
PDF