Issue #1152
closed
Pulp's Celery processes uses PLAIN sasl mech unexpectedly with Qpid
Description
This bug causes an unexpected, random deadlock condition which can cause pulp_workers, pulp_resource_manager, or pulp_celerybeat to deadlock. This will likely cause tasks to appear stuck in the 'Waiting' state. You can confirm if one of your stuck processes was affected by this if you see a log line like the following when you restart. You may notice restart is also sluggish.
Please enter your password: Please enter your password: Please enter your password:
This is tracking the inclusion of an upstream Kombu fix. https://github.com/celery/kombu/issues/498
Along with the inclusion of the sasl mech selection change in kombu, the default broker string used by the [tasks] section of server.conf needs to have its 'guest@' removed. A Pulp release note needs to be made for this conf file change.
Added by bmbouter over 9 years ago
Added by bmbouter over 9 years ago
Revision d73f4753 | View on GitHub
Adjusts Qpid broker string and includes Kombu SASL fixes.
This includes all the patches from before consolidated into one patch and also includes the fix for https://github.com/celery/kombu/issues/498
The pulp broker string for the [tasks] section is adjusted to not specify 'guest' as the default username. This will force Kombu to use ANONYMOUS auth by default.
Updated by bmbouter over 9 years ago
- Status changed from NEW to POST
PR available at: https://github.com/pulp/pulp/pull/1993
Updated by bmbouter over 9 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|d73f4753d9c9a6a2867e12928b214d489081713a.
Updated by bmbouter over 9 years ago
- Platform Release set to 2.6.4
QE, to verify this you should do it on EL7:
1. Setup a default installation of Pulp with Qpid
2. Verify that python-kombu-3.0.24-9 is installed.
3. Ensure that qpidd.conf is completely empty and restart Qpid to ensure those settings are in place
4. Start Pulp and sanity check that it is working (ie: sync+publish)
5. With Pulp running, list the connections by running `qpid-stat -c`. You may need to install qpid-tools to run `qpid-stat`.
6. Verify that all of the connections show they are using ANONYMOUS. Ensure none of them are using PLAIN.
Updated by dkliban@redhat.com over 9 years ago
- Platform Release changed from 2.6.4 to 2.6.5
Updated by dkliban@redhat.com about 9 years ago
- Status changed from MODIFIED to 5
Updated by dkliban@redhat.com about 9 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
Adjusts Qpid broker string and includes Kombu SASL fixes.
This includes all the patches from before consolidated into one patch and also includes the fix for https://github.com/celery/kombu/issues/498
The pulp broker string for the [tasks] section is adjusted to not specify 'guest' as the default username. This will force Kombu to use ANONYMOUS auth by default.
https://pulp.plan.io/issues/1152 closes #1152