Project

Profile

Help

Issue #4929

Pulp 2.18 having issues to start workers celery and communication to rabbitmq

Added by ymadav over 1 year ago. Updated about 1 month ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
4. Urgent
Version:
Platform Release:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Team,

We upgraded pulp version from 2.12 to 2.18 and all the services are up and running but,worker services in worker server are not able to start and it is failing to communicate to rabbitmq server.we have below package versions installed in worker for celery and pulp,also posting the pulp errors.Please help us to fix this as this is production we are running out of the time.uploaded the message file for reference,please let me know if anything else is required.

We have earlier python2-celery-4.2.1-3.el7.noarch version installed,even with that we were unable to start the services,so using python2-celery-4.0.2-7.el7.noarch.Server is running with RHEL 7.6

Mostly i see these errors

Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from billiard.compat import get_fdmax, close_open_fds
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: ImportError: cannot import name get_fdmax

Installed packaged
rpm qa | grep -i celery
python2-celery-4.0.2-7.el7.noarch
[ymadav@ip-10-12-111-237 ~]$ rpm -qa | egrep -i "amqp|celery|billiard|kombu"
python2-celery-4.0.2-7.el7.noarch
python2-kombu-4.0.2-11.el7.noarch
python-gofer-amqp-2.12.5-1.el7.noarch
python2-amqp-2.2.2-5.el7.noarch
python2-billiard-3.5.0.3-4.el7.x86_64
[ymadav@ip-10-12-111-237 ~]$ rpm -qa | grep -i pulp
*
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-rpm-common-2.18.1-1.el7.noarch
pulp-rpm-plugins-2.18.1-1.el7.noarch
python-pulp-puppet-common-2.18.1-1.el7.noarch
pulp-selinux-2.18.1-2.el7.noarch
pulp-puppet-tools-2.18.1-1.el7.noarch
python-pulp-manifest-2.16.4-1.el7.noarch
python-pulp-docker-common-3.2.2-1.el7.noarch
pulp-docker-plugins-3.2.2-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
python-pulp-deb-common-1.8.0-1.el7.noarch
python-pulp-repoauth-2.18.1-2.el7.noarch
python2-solv-0.6.34-2.pulp.el7.x86_64
pulp-deb-plugins-1.8.0-1.el7.noarch
python-pulp-ostree-common-1.4.0-1.el7.noarch
pulp-puppet-plugins-2.18.1-1.el7.noarch
python-pulp-oid_validation-2.18.1-2.el7.noarch
pulp-server-2.18.1-2.el7.noarch
python-pulp-common-2.18.1-2.el7.noarch
libsolv-0.6.34-2.pulp.el7.x86_64
pulp-ostree-plugins-1.4.0-1.el7.noarch

sudo systemctl list-units | grep -i pulp
● pulp_worker-0.service                                                               loaded failed failed    Pulp Worker #0
● pulp_worker-1.service                                                               loaded failed failed    Pulp Worker #1
● pulp_worker-2.service                                                               loaded failed failed    Pulp Worker #2
● pulp_worker-3.service                                                               loaded failed failed    Pulp Worker #3
 pulp_workers.service                                                                loaded active exited    Pulp Celery Workers

Jun 07 04:17:21 ip-10-12-111-237. celery[319]: File "/usr/lib/python2.7/site-packages/celery/utils/log.py", line 19, in <module>
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from .term import colored
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: File "/usr/lib/python2.7/site-packages/celery/utils/term.py", line 11, in <module>
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from celery.platforms import isatty
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: File "/usr/lib/python2.7/site-packages/celery/platforms.py", line 21, in <module>
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: from billiard.compat import get_fdmax, close_open_fds
Jun 07 04:17:21 ip-10-12-111-237. celery[319]: ImportError: cannot import name get_fdmax
Jun 07 04:17:21 ip-10-12-111-237. systemd[1]: pulp_worker-0.service: main process exited, code=exited, status=1/FAILURE
Jun 07 04:17:21 ip-10-12-111-237. systemd[1]: Unit pulp_worker-0.service entered failed state.
Jun 07 04:17:21 ip-10-12-111-237. systemd[1]: pulp_worker-0.service failed.

Thanks,
-Yash

clipboard-201906070629-xt6og.png (33.8 KB) clipboard-201906070629-xt6og.png ymadav, 06/07/2019 06:29 AM
messages (4.67 MB) messages ymadav, 06/07/2019 06:33 AM
clipboard-201910061437-p0te4.png (2.73 KB) clipboard-201910061437-p0te4.png Anonymous, 10/06/2019 11:38 AM
250
250

History

#1 Updated by ymadav over 1 year ago

daviddavis can you please help on this.

#2 Updated by amacdona@redhat.com over 1 year ago

  • Project changed from Infrastructure to Pulp

#3 Updated by daviddavis over 1 year ago

  • Status changed from NEW to CLOSED - WORKSFORME

This is an environment problem and not a code defect. Would you mind emailing the mailing list or visiting #pulp on freenode IRC?

Please register to edit this issue

Also available in: Atom PDF