Project

Profile

Help

Issue #1039

Waiting reaper tasks

Added by Ben.Stanley over 5 years ago. Updated 9 months ago.

Status:
CLOSED - NOTABUG
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.6.1
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

My pulp server seems to accumulate Waiting reaper tasks.

What are they waiting for?

[root@bumblebee ~]# pulp-admin tasks list | grep Running | wc -l
0
[root@bumblebee ~]# pulp-admin tasks list | grep Waiting | wc -l
13

[root@bumblebee ~]# pulp-admin tasks list | less
------------------------------------------------------------------------------
Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: a688c1b8-71ec-4230-b60f-a72e77f302ff

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 798d04e1-3f25-45d9-96c0-487a01ff6be6

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 4e3f7ba9-34b7-443f-9cb9-8e3345d7b9a2

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 7a050432-7a06-426e-8eca-5c0124c7b722

Operations: reaper
Resources:
State: Successful
Start Time: 2015-06-02T18:50:59Z
Finish Time: 2015-06-02T18:50:59Z
Task Id: 2278c944-3392-4b6f-954d-0097ed1ee0cf

Operations: publish
Resources: elrepo-7-x86_64-test1 (repository)
State: Successful
Start Time: 2015-06-03T00:30:54Z
Finish Time: 2015-06-03T00:30:54Z
Task Id: 4582a809-4304-4587-acd8-ddfcfddfb4a0

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 2a56ff42-27ac-4d44-ba6c-d29e989d4ffe

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 9ccce189-ccf0-4baa-ae3a-8649ecaff807

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 9091d8f1-b124-4b19-a5f3-943706e8f82f

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 8fd902c1-0de8-465a-8247-e0d9152e8b2d

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: c4bf7f08-9863-423c-88cc-efcaa4e4ecc1

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: a56433e0-bfcf-4262-a654-35e08926ff2b

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 524fba1f-298e-4472-9f17-b54414a0262b

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: b719da7f-2634-4224-9a7a-c6307e2fe959

Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 872b859c-a885-40ed-8d79-31e1b63f3b24
------------------------------------------------------------------------------

Searching the log file for the last task finds only
------------------------------------------------------------------------------
Jun 8 22:51:04 bumblebee pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[872b859c-a885-40ed-8d79-31e1b63f3b24]
------------------------------------------------------------------------------

There are some successful reaper tasks too:
[root@bumblebee ~]# pulp-admin tasks list | grep reaper | wc -l
37

History

#1 Updated by mhrivnak over 5 years ago

  • Priority changed from Normal to High
  • Triaged changed from No to Yes

#2 Updated by bmbouter over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

#3 Updated by bmbouter over 5 years ago

Ben, can you provide the output from `ps -awfux | grep celery` and also `rpm -qa | grep pulp`. Thanks.

#4 Updated by Ben.Stanley over 5 years ago

Unfortunately, I have had to clear this erroneous state to investigate [[https://pulp.plan.io/issues/1008]].

[root@bumblebee repodata]# pulp-admin tasks list | grep -e Waiting | wc -l
0
[root@bumblebee repodata]# pulp-admin tasks list | grep -e Running | wc -l
0

So there are currently no reaper tasks running or waiting.

I will add more information to this issue if the state is reproduced.

#5 Updated by bmbouter almost 5 years ago

  • Status changed from ASSIGNED to CLOSED - NOTABUG
  • Assignee deleted (bmbouter)

@Ben.Stanley Have you re-observed this issue recently? It's been several months and several small improvements have been made to the tasking system since then. Please reopen if this is still an issue for you.

#6 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF