Issue #1330
closedcelery worker memory leak and high CPU after restarting qpidd many times
Description
To reproduce:
1. Start Pulp normally. I reproduced this on 2.6.4.
2. While Pulp is running, restart qpidd very often (every few seconds). I do that with this shell script:
while true
do sudo service qpidd restart
sleep 1.1
sleep $(($((RANDOM))%2))
done
3. Observe that over a few minutes the CPU will grow. You can watch this by periodically listing the processes and looking at the CPU column. I do this with
ps -awfux | grep celery
4. Once your processes are consuming lots of CPU for no apparent reason, stop the qpidd restart script and ensure that qpidd is started.
5. Observe that your CPU never goes back down even though Pulp is not doing any work and qpidd is running normally.
6. Despair
You can recover from this by restarting the celery processes. The original reporter who told me about this also indicated there was a memory leak, but I haven't confirmed this.
Related issues
Updated by bmbouter almost 9 years ago
I am not actively working on this so I'm setting back to NEW.
Updated by bmbouter almost 9 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
bmbouter)
Updated by bmbouter almost 8 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by bmbouter almost 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to bmbouter
Updated by bizhang over 7 years ago
- Related to Story #2632: As a developer I want to reevaluate worker issues to see if they have been resolved by moving from Celery3 to Celery4 added
Updated by mhrivnak over 7 years ago
In sprint planning we decided to wait on this issue, and let the upgrade to celery 4 fix it for us. We will try to reproduce on the new celery 4 stack, but expect it to be resolved.
See story #2632
Updated by bmbouter over 7 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
bmbouter) - Sprint/Milestone deleted (
34)
Updated by dkliban@redhat.com over 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
- Sprint Candidate set to No
This is not going to be fixed for Pulp 2. Pulp 3 does not rely on celery or qpid.