Project

Profile

Help

Issue #1330

closed

celery worker memory leak and high CPU after restarting qpidd many times

Added by bmbouter over 7 years ago. Updated over 3 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
2.6.4
Platform Release:
OS:
Fedora 22
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

To reproduce:

1. Start Pulp normally. I reproduced this on 2.6.4.
2. While Pulp is running, restart qpidd very often (every few seconds). I do that with this shell script:

while true
do sudo service qpidd restart
sleep 1.1
sleep $(($((RANDOM))%2))
done

3. Observe that over a few minutes the CPU will grow. You can watch this by periodically listing the processes and looking at the CPU column. I do this with

ps -awfux | grep celery

4. Once your processes are consuming lots of CPU for no apparent reason, stop the qpidd restart script and ensure that qpidd is started.
5. Observe that your CPU never goes back down even though Pulp is not doing any work and qpidd is running normally.
6. Despair

You can recover from this by restarting the celery processes. The original reporter who told me about this also indicated there was a memory leak, but I haven't confirmed this.


Related issues

Related to Pulp - Story #2632: As a developer I want to reevaluate worker issues to see if they have been resolved by moving from Celery3 to Celery4CLOSED - WONTFIX

Actions
Actions #1

Updated by mhrivnak over 7 years ago

  • Triaged changed from No to Yes
Actions #2

Updated by bmbouter about 7 years ago

I am not actively working on this so I'm setting back to NEW.

Actions #3

Updated by bmbouter about 7 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)
Actions #5

Updated by bmbouter about 6 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #6

Updated by ipanova@redhat.com about 6 years ago

  • Sprint/Milestone set to 31
Actions #7

Updated by bmbouter about 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
Actions #8

Updated by mhrivnak about 6 years ago

  • Sprint/Milestone changed from 31 to 32
Actions #9

Updated by mhrivnak almost 6 years ago

  • Sprint/Milestone changed from 32 to 33
Actions #10

Updated by mhrivnak almost 6 years ago

  • Sprint/Milestone changed from 33 to 34
Actions #11

Updated by bizhang almost 6 years ago

  • Related to Story #2632: As a developer I want to reevaluate worker issues to see if they have been resolved by moving from Celery3 to Celery4 added
Actions #12

Updated by mhrivnak almost 6 years ago

In sprint planning we decided to wait on this issue, and let the upgrade to celery 4 fix it for us. We will try to reproduce on the new celery 4 stack, but expect it to be resolved.

See story #2632

Actions #13

Updated by bmbouter almost 6 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)
  • Sprint/Milestone deleted (34)
Actions #14

Updated by rchan almost 5 years ago

  • Sprint Candidate deleted (Yes)
Actions #15

Updated by bmbouter almost 4 years ago

  • Tags Pulp 2 added
Actions #16

Updated by dkliban@redhat.com over 3 years ago

  • Status changed from NEW to CLOSED - WONTFIX
  • Sprint Candidate set to No

This is not going to be fixed for Pulp 2. Pulp 3 does not rely on celery or qpid.

Also available in: Atom PDF