Actions
Issue #4387
closedEpel is shipping a new version of celery which doesn't work with Pulp
Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 49
Quarter:
Description
Problem¶
On the latest build of 2.18.1b1 for 2019-02-06, the following test has a failure when killing amqp that the service never recovers on non-fips boxes. FIPS enabled boxes do not have this problem and also have a different version of python-amqp installed:
RPM diff out¶
[root@rhel76-2181b1 ~]# diff all_fips.out all_nonfips.out
66,67d65
< dracut-fips-033-554.el7.x86_64
< dracut-fips-aesni-033-554.el7.x86_64
89a88
> fpaste-0.3.7.4.1-2.el7.noarch
124d122
< hmaccalc-0.9.13-4.el7.x86_64
399c397
< python2-amqp-2.2.2-3.el7.noarch
---
> python2-amqp-2.4.0-1.el7.noarch
401,402c399,400
< python2-billiard-3.5.0.3-3.el7.x86_64
< python2-celery-4.0.2-5.el7.noarch
---
> python2-billiard-3.5.0.5-1.el7.x86_64
> python2-celery-4.2.1-3.el7.noarch
407c405
< python2-django-1.11.17-1.el7.noarch
---
> python2-django-1.11.18-1.el7.noarch
410c408
< python2-kombu-4.0.2-9.el7.noarch
---
> python2-kombu-4.2.2-1.el7.noarch
418,419c416
< python2-vine-1.1.3-4.el7.noarch
< python-anyjson-0.3.3-3.el7.noarch
---
> python2-vine-1.2.0-1.el7.noarch
434a432
> python-django-bash-completion-1.11.18-1.el7.noarch
555a554
> shark-0.1-1.noarch
562a562
> stork-0.12-2.noarch
583a584
> whale-0.2-1.noarch
592d592
< yum-plugin-priorities-1.1.31-50.el7.noarch
[root@rhel76-2181b1 ~]#
Test Failure¶
an 29 16:51:37 localhost.localdomain systemd[1]: Stopping An AMQP message broker daemon....
Jan 29 16:51:37 localhost.localdomain pulp[31599]: gofer.messaging.adapter.qpid.reliability:WARNING: connection aborted
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) consumer: Connection to broker lost. Trying to re-establish the connection...
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) Traceback (most recent call last):
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) blueprint.start(self)
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) step.start(parent)
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 593, in start
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) c.loop(*c.loop_args())
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 121, in synloop
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) connection.drain_events(timeout=2.0)
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) return self.transport.drain_events(self.connection, **kwargs)
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1693, in drain_events
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) receiver = self.session.next_receiver(timeout=timeout)
Jan 29 16:51:37 localhost.localdomain pulp[31580]: celery.worker.consumer.consumer:WARNING: (31580-24256) File "<string>", line 6, in next_receiver
Test Recation¶
Run the following tests from Pulp-2-Test
- pytest -sv pulp_2_tests/tests/rpm/api_v2/test_service_resiliency.py
Files
Related issues
Actions
Changing Ansible Installer priorities with current 2.18.1 definition
To solve an EPEL version problem with the python-celery stack, the final solution during build has changed the EPOCH on NEVRA for those RPMs to 10:
The resulting files needed to be updated to:
The end state of each of those upgrade permutations leave the box in the same RPM state without the end-user to have to play with yum or repository priorities.
Removing the pulp-priority role as it is now deprecated with these changes.
The included, tested changes are the output of the solution for CI.
refs #4387