Project

Profile

Help

Issue #1564

closed

Unacked messages in resource_manager queue

Added by bmbouter about 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.7.0
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

After executing a "heavy" workload, the resource_manager queue may have unacked messages in it. To reproduce you can:

1. Execute the heavy workload and let it finish. For instance you could sync many repos and publish them.

2. List the Qpid queue statistics with:

qpid-stat -q

3. Observe a output like the following:

queue                                                                             dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind 
=================================================================================================================================================================
<snip>
resource_manager                                                                  Y                     19   338    319    33.4k   567k     534k        1     2
<snip>

4. Observe that all Pulp tasks have completed with state finished, and yet there are still messages in the 'msg' column. 19 in the example above. You can verify all Pulp tasks as being completed by connecting to mongo, switching to the pulp_database, and running the following command. If all pulp tasks are finished that query will return 0.

db.task_status.find({ state:{ $ne: "finished" } } ).count()

Note: This has been observed on both EL6 and EL7, and it may be observable on Fedoras also.

Any of the unacked messages will be "rerun" when the resource_manager restarts because the message broker will redeliver them. This is a user noticeable defect.

Actions #2

Updated by bmbouter about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
Actions #3

Updated by bmbouter about 8 years ago

I've been able to reproduce this in my dec environment.

First I reduce the number of workers I have to 1.

Then I create two repos:

pulp-admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/pulp/pulp/demo_repos/zoo/
pulp-admin rpm repo create --repo-id zoo2 --feed https://repos.fedorapeople.org/pulp/pulp/demo_repos/zoo2/

Then I make a script which will sync them both:

[vagrant@dev scripts]$ cat sync_zoo1_zoo2.sh 
http --verify no -v --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo/actions/sync/
http --verify no -v --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo2/actions/sync/

Then I run that script 15 times quickly in a bash loop:

for i in {1..15}; do sh sync_zoo1_zoo2.sh; done

Then observe using `qpid-stat -q` that there are unacked messages in the resource_manager queue after all of the syncs have completed.

Actions #4

Updated by jortel@redhat.com about 8 years ago

  • Priority changed from Normal to High
  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes
Actions #5

Updated by jortel@redhat.com about 8 years ago

  • Platform Release deleted (2.8.0)
Actions #6

Updated by bmbouter about 8 years ago

This is a bug in Celery. Upstream bug filed here: https://github.com/celery/celery/issues/3019

Added by bmbouter about 8 years ago

Revision 0d2de5d5 | View on GitHub

Automatic commit of package [python-kombu] minor release [3.0.24-11.pulp].

Added by bmbouter about 8 years ago

Revision 83bd85c4 | View on GitHub

Adds python-kombu-3.0.24-11 to external_deps.json

https://pulp.plan.io/issues/1564 re #1564

Added by bmbouter about 8 years ago

Revision 83bd85c4 | View on GitHub

Adds python-kombu-3.0.24-11 to external_deps.json

https://pulp.plan.io/issues/1564 re #1564

Actions #7

Updated by bmbouter about 8 years ago

  • Status changed from ASSIGNED to POST
Actions #8

Updated by bmbouter about 8 years ago

  • Status changed from POST to MODIFIED
  • Platform Release set to 2.7.2
Actions #9

Updated by bmbouter about 8 years ago

  • Description updated (diff)
Actions #10

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from MODIFIED to 5
Actions #12

Updated by semyers almost 8 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
  • Version set to 2.8.0
Actions #13

Updated by semyers almost 8 years ago

  • Version changed from 2.8.0 to 2.7.0
  • Platform Release changed from 2.7.2 to 2.8.0

Updated the wrong field. Sorry 'bout that.

Actions #15

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF