Project

Profile

Help

Issue #1152

Pulp's Celery processes uses PLAIN sasl mech unexpectedly with Qpid

Added by bmbouter over 6 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.6.5
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

This bug causes an unexpected, random deadlock condition which can cause pulp_workers, pulp_resource_manager, or pulp_celerybeat to deadlock. This will likely cause tasks to appear stuck in the 'Waiting' state. You can confirm if one of your stuck processes was affected by this if you see a log line like the following when you restart. You may notice restart is also sluggish.

Please enter your password: Please enter your password: Please enter your password:

This is tracking the inclusion of an upstream Kombu fix. https://github.com/celery/kombu/issues/498

Along with the inclusion of the sasl mech selection change in kombu, the default broker string used by the [tasks] section of server.conf needs to have its 'guest@' removed. A Pulp release note needs to be made for this conf file change.

Associated revisions

Revision d73f4753 View on GitHub
Added by bmbouter over 6 years ago

Adjusts Qpid broker string and includes Kombu SASL fixes.

This includes all the patches from before consolidated into one patch and also includes the fix for https://github.com/celery/kombu/issues/498

The pulp broker string for the [tasks] section is adjusted to not specify 'guest' as the default username. This will force Kombu to use ANONYMOUS auth by default.

https://pulp.plan.io/issues/1152 closes #1152

Revision d73f4753 View on GitHub
Added by bmbouter over 6 years ago

Adjusts Qpid broker string and includes Kombu SASL fixes.

This includes all the patches from before consolidated into one patch and also includes the fix for https://github.com/celery/kombu/issues/498

The pulp broker string for the [tasks] section is adjusted to not specify 'guest' as the default username. This will force Kombu to use ANONYMOUS auth by default.

https://pulp.plan.io/issues/1152 closes #1152

History

#2 Updated by bmbouter over 6 years ago

  • Status changed from NEW to POST

#3 Updated by bmbouter over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#4 Updated by bmbouter over 6 years ago

  • Platform Release set to 2.6.4

QE, to verify this you should do it on EL7:

1. Setup a default installation of Pulp with Qpid
2. Verify that python-kombu-3.0.24-9 is installed.
3. Ensure that qpidd.conf is completely empty and restart Qpid to ensure those settings are in place
4. Start Pulp and sanity check that it is working (ie: sync+publish)
5. With Pulp running, list the connections by running `qpid-stat -c`. You may need to install qpid-tools to run `qpid-stat`.
6. Verify that all of the connections show they are using ANONYMOUS. Ensure none of them are using PLAIN.

#5 Updated by mhrivnak over 6 years ago

  • Triaged changed from No to Yes

#6 Updated by bmbouter over 6 years ago

  • Assignee set to bmbouter

#7 Updated by dkliban@redhat.com over 6 years ago

  • Platform Release changed from 2.6.4 to 2.6.5

#8 Updated by bmbouter over 6 years ago

  • Description updated (diff)

#9 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5

#10 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#13 Updated by bmbouter almost 3 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF