Task #3340
closed
Task #3291: Switch Pulp to AMQP 1.0
Testing of new Kombu stack
0%
Description
There are many feature aspects of the Qpid transport for Kombu that Pulp relies on. Recently this Qpid transport for Kombu was entirely rewritten. Before we enable it for either Pulp2 users ( issue 3134 ) or Pulp3 users ( issue 3292) we need to ensure that everything Pulp relies on is working.
As the author of the Kombu changes [0], I've already hand-tested all of this, so now it's time for independant testing. Here are all of the high level things that need to work:
Connect and Reconnect Testing¶
- Verify that if Pulp is started but Qpid is stopped, that when Qpid does start and become available that Pulp connects and is fully working. A basic sync will prove the "fully working part".
- Verify that "reconnect support" works. Have Pulp already connected to Qpid and functioning correctly. Then stop qpidd, but leave all Pulp processes running. After say
Authentication testing¶
- Verify that if qpidd requires no authentication, that Pulp can connect to qpidd and function properly.
- Verify that if qpidd requires SASL authentication, that Pulp can be correctly configured and function properly.
- Verify that if qpidd requires the client to present a certificate via EXTERNAL auth that Pulp can be correctly configured to present that certificate, can connect, and can function properly.
- Verify that the Pulp can be configured to require the server to present a trusted certificate. Specifically Pulp needs to be configured to trust the CA that qpidd is presenting. In this configuration verify that Pulp can function properly.
- Verify that mutual authentication works. Specifically this is the client requiring a trusted cert from qpidd, and qpidd requiring a trusted cert from the client. These are the above two test scenarios only tested together.
- Verify that an incorrect authentication configuration does not try to reconnect. Reconnecting with the same incorrect credentials needs to not happen.
- Verify that when celery_require_ssl is False that Pulp can connect to qpidd without TLS
- Verify that when celery_require_ssl is True that Pulp can not connect to qpidd without TLS
Queue, Exchange, and Bindings settings¶
- Verify that all of the same queues are created and have identical properties when using the new Kombu transport versus the older one.
- Verify that all of the same exchanges are created and have identical properties when using the new Kombu transport versus the older one.
- Verify that all of the same bindings are created and have identical properties when using the new Kombu transport versus the older one.
Files
Updated by bmbouter almost 7 years ago
- Blocks Task #3134: Switch Pulp 3 installer to use qpid by default, not rabbitmq added
Updated by bmbouter almost 7 years ago
- Subject changed from Testing of new Kombu stack with Pulp3 to Testing of new Kombu stack
Updated by pthomas@redhat.com almost 7 years ago
I am getting ready to test this on Pulp2. Could you please give me some guidelines on how to go about doing the upgrade.
Updated by bmbouter almost 7 years ago
It involves checking out two custom code repositories, ensuring you have the newest Celery+Kombu stack (4.x), installing proton and the proton qpid bindings.
One option is to look at the Ansible I wrote here: https://github.com/pulp/devel/blob/3.0-dev/ansible/roles/kombu_fixup/tasks/main.yml
For the custom code checkouts, you need the HEAD of these two branches installed:
https://github.com/bmbouter/kombu/tree/624-convert-qpid-to-amqp-1.0
https://github.com/bmbouter/qpid-cpp/tree/qpid-for-pulp
Before ^ ansible code, I was using the following commands. YMMV because this is for pulp3, but pulp2 is not installed in a virtualenv and also already has qpid enabled so you don't have to disable rabbitmq.
pip uninstall kombu
cd ../kombu;python setup.py develop # this installs the custom kombu branch
sudo dnf install python-qpid-proton python3-devel gcc qpid-cpp-server
pip install python-qpid-proton
cd /home/vagrant/devel/qpid-cpp/management/python;python setup.py install # this installs the custom qpid-cpp branch
Thanks for the testing. Post any questions you have here and I can answer them.
Updated by pthomas@redhat.com over 6 years ago
- File amqp_upgrade.txt amqp_upgrade.txt added
I tried the upgrade and have experienced some issues.
I will try to summarize the steps and issues that I have encountered.
1. pip uninstall kombu
2. pip install "git+https://github.com/bmbouter/kombu.git@624-convert-qpid-to-amqp-1.0#egg=kombu"
3. sudo yum install python-qpid-proton python2-devel gcc qpid-cpp-server
4. pip install python-qpid-proton
5. pip install "git+https://github.com/bmbouter/qpid-cpp.git@qpid-for-pulp#subdirectory=management/python"
Collecting git+https://github.com/bmbouter/qpid-cpp.git@qpid-for-pulp#subdirectory=management/python
Cloning https://github.com/bmbouter/qpid-cpp.git (to qpid-for-pulp) to /tmp/pip-R1A3Yb-build
Requirement already satisfied (use --upgrade to upgrade): qpid-tools==1.37.0 from git+https://github.com/bmbouter/qpid-cpp.git@qpid-for-pulp#subdirectory=management/python in /usr/lib/python2.7/site-packages
Requirement already satisfied: qpid-python>=0.26 in /usr/lib/python2.7/site-packages (from qpid-tools==1.37.0)
6. pip install "git+https://github.com/bmbouter/qpid-cpp.git@qpid-for-pulp#subdirectory=management/python" --upgrade
Collecting git+https://github.com/bmbouter/qpid-cpp.git@qpid-for-pulp#subdirectory=management/python
Cloning https://github.com/bmbouter/qpid-cpp.git (to qpid-for-pulp) to /tmp/pip-44_Avx-build
Requirement already up-to-date: qpid-python>=0.26 in /usr/lib/python2.7/site-packages (from qpid-tools==1.37.0)
Installing collected packages: qpid-tools
Found existing installation: qpid-tools 1.37.0
DEPRECATION: Uninstalling a distutils installed project (qpid-tools) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling qpid-tools-1.37.0:
Successfully uninstalled qpid-tools-1.37.0
Running setup.py install for qpid-tools ... done
Successfully installed qpid-tools-1.37.0
After this, I restarted the services
And this resulted in
Mar 01 11:45:13 mgmt8.rhq.lab.eng.bos.redhat.com celery[6865]: raise ImproperlyConfigured('Celery 4.x requires Django 1.8 or later.')
Mar 01 11:45:13 mgmt8.rhq.lab.eng.bos.redhat.com celery[6865]: celery.exceptions.ImproperlyConfigured: Celery 4.x requires Django 1.8 or later.
Mar 01 11:45:13 mgmt8.rhq.lab.eng.bos.redhat.com systemd[1]: pulp_celerybeat.service: main process exited, code=exited, status=1/FAILURE
Mar 01 11:45:13 mgmt8.rhq.lab.eng.bos.redhat.com systemd[1]: Unit pulp_celerybeat.service entered failed state.
Mar 01 11:45:13 mgmt8.rhq.lab.eng.bos.redhat.com systemd[1]: pulp_celerybeat.service failed.
Then I upgraded Django
[root@mgmt8 ~]# pip install django --upgrade
Collecting django
Downloading Django-1.11.10-py2.py3-none-any.whl (6.9MB)
100% |████████████████████████████████| 7.0MB 93kB/s
Requirement already up-to-date: pytz in /usr/lib/python2.7/site-packages (from django)
Installing collected packages: django
Found existing installation: Django 1.6.11.6
DEPRECATION: Uninstalling a distutils installed project (django) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling Django-1.6.11.6:
Successfully uninstalled Django-1.6.11.6
Successfully installed django-1.11.10
I restarted the service and seeing the errors (attached file) in the journalctl
Updated by dalley over 6 years ago
@preethi, this is a really good find, because support for Django 1.6 expired in April 2015 (!), and support for Django 1.8, which is an LTS version, expires in 3 weeks (April 1, 2018).
We need to upgrade Django to the 1.11 LTS release and resolve those issues.
Updated by dalley over 6 years ago
- Blocks deleted (Task #3134: Switch Pulp 3 installer to use qpid by default, not rabbitmq)
Updated by bmbouter over 6 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp3 is now on RQ so Pulp won't benefit from the additional testing of this Kombu stack since it will never use it. More info on the RQ transition is here: https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
.