Pulp

Team,

We upgraded pulp version from 2.12 to 2.18 and all the services are up and running but,worker services in worker server are not able to start and it is failing to communicate to rabbitmq server.we have below package versions installed in worker for celery and pulp,also posting the pulp errors.Please help us to fix this as this is production we are running out of the time.uploaded the message file for reference,please let me know if anything else is required.

We have earlier python2-celery-4.2.1-3.el7.noarch version installed,even with that we were unable to start the services,so using python2-celery-4.0.2-7.el7.noarch.Server is running with RHEL 7.6

Mostly i see these errors

Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from billiard.compat import get_fdmax, close_open_fds
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: ImportError: cannot import name get_fdmax

Installed packaged
rpm qa | grep -i celery
python2-celery-4.0.2-7.el7.noarch
[ymadav@ip-10-12-111-237 ~]$ rpm -qa | egrep -i "amqp|celery|billiard|kombu"
python2-celery-4.0.2-7.el7.noarch
python2-kombu-4.0.2-11.el7.noarch
python-gofer-amqp-2.12.5-1.el7.noarch
python2-amqp-2.2.2-5.el7.noarch
python2-billiard-3.5.0.3-4.el7.x86_64
[ymadav@ip-10-12-111-237 ~]$ rpm -qa | grep -i pulp*
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-rpm-common-2.18.1-1.el7.noarch
pulp-rpm-plugins-2.18.1-1.el7.noarch
python-pulp-puppet-common-2.18.1-1.el7.noarch
pulp-selinux-2.18.1-2.el7.noarch
pulp-puppet-tools-2.18.1-1.el7.noarch
python-pulp-manifest-2.16.4-1.el7.noarch
python-pulp-docker-common-3.2.2-1.el7.noarch
pulp-docker-plugins-3.2.2-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
python-pulp-deb-common-1.8.0-1.el7.noarch
python-pulp-repoauth-2.18.1-2.el7.noarch
python2-solv-0.6.34-2.pulp.el7.x86_64
pulp-deb-plugins-1.8.0-1.el7.noarch
python-pulp-ostree-common-1.4.0-1.el7.noarch
pulp-puppet-plugins-2.18.1-1.el7.noarch
python-pulp-oid_validation-2.18.1-2.el7.noarch
pulp-server-2.18.1-2.el7.noarch
python-pulp-common-2.18.1-2.el7.noarch
libsolv-0.6.34-2.pulp.el7.x86_64
pulp-ostree-plugins-1.4.0-1.el7.noarch

sudo systemctl list-units | grep -i pulp
● pulp_worker-0.service                                                               loaded failed failed    Pulp Worker #0
● pulp_worker-1.service                                                               loaded failed failed    Pulp Worker #1
● pulp_worker-2.service                                                               loaded failed failed    Pulp Worker #2
● pulp_worker-3.service                                                               loaded failed failed    Pulp Worker #3
 pulp_workers.service                                                                loaded active exited    Pulp Celery Workers

Jun 07 04:17:21 ip-10-12-111-237. celery[319]: File "/usr/lib/python2.7/site-packages/celery/utils/log.py", line 19, in <module>
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from .term import colored
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: File "/usr/lib/python2.7/site-packages/celery/utils/term.py", line 11, in <module>
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from celery.platforms import isatty
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: File "/usr/lib/python2.7/site-packages/celery/platforms.py", line 21, in <module>
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from billiard.compat import get_fdmax, close_open_fds
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: ImportError: cannot import name get_fdmax
Jun 07 04:17:21 ip-10-12-111-237. systemd[1]: pulp_worker-0.service: main process exited, code=exited, status=1/FAILURE
Jun 07 04:17:21 ip-10-12-111-237. systemd[1]: Unit pulp_worker-0.service entered failed state.
Jun 07 04:17:21 ip-10-12-111-237. systemd[1]: pulp_worker-0.service failed.

Thanks,
-Yash