Issue #2586: Pulp workers/beat/resource_manager go missing - Pulp

Actions

Send by e-mail Copy link

Issue #2586

closed

Pulp workers/beat/resource_manager go missing

Added by ehelms@redhat.com over 6 years ago. Updated over 4 years ago.

Status:

CLOSED - NOTABUG

Priority:

Urgent

Assignee:

daviddavis

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Sprint 15

Quarter:

Description

Running on Pulp 2.10.3, I see the following error messages running this setup:

clone https://github.com/Katello/forklift
vagrant up centos7-katello-nightly

If you do not see the error messages initially:

vagrant ssh centos7-katello-nightly
cd /
/vagrant/forklift/bats/bootstrap_vagrant.sh

cd /vagrant/forklift/bats
bats fb-install-katello.bats fb-content-katello.bats

Feb 13 20:45:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:46:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:47:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:48:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:49:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:50:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.

Actions

Copy link

Updated by bizhang over 6 years ago

Priority changed from Normal to Urgent
Sprint/Milestone set to 33
Triaged changed from No to Yes

Actions

Copy link

Updated by daviddavis over 6 years ago

Assignee set to daviddavis

Actions

Copy link

Updated by daviddavis over 6 years ago

Status changed from NEW to ASSIGNED

Actions

Copy link

Updated by daviddavis over 6 years ago

It looks like pulp is having problems connecting to qpid:

Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136) consumer: Connection to broker lost. Trying to re-establish the connection...
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136) Traceback (most recent call last):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     blueprint.start(self)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     step.start(parent)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376) consumer: Connection to broker lost. Trying to re-establish the connection...
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     c.loop(*c.loop_args())
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376) Traceback (most recent call last):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     next(loop)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     blueprint.start(self)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     cb(*cbargs)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     step.start(parent)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1493, in on_readable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     self.drain_events(connection)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1647, in drain_events
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     c.loop(*c.loop_args())
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     receiver = self.session.next_receiver(timeout=timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "<string>", line 6, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     next(loop)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 777, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     cb(*cbargs)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     if self._ecwait(lambda: self.incoming, timeout):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 57, in _ecwait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1493, in on_readable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     result = self._ewait(lambda: self.closed or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     self.drain_events(connection)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1647, in drain_events
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     receiver = self.session.next_receiver(timeout=timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "<string>", line 6, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     self.check_error()
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 777, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     if self._ecwait(lambda: self.incoming, timeout):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     raise e
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 57, in _ecwait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136) ConnectionError: [Errno 101] Network is unreachable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     result = self._ewait(lambda: self.closed or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     self.check_error()
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     raise e
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376) ConnectionError: [Errno 101] Network is unreachable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984) consumer: Connection to broker lost. Trying to re-establish the connection...
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984) Traceback (most recent call last):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     blueprint.start(self)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     step.start(parent)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     c.loop(*c.loop_args())
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     next(loop)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     cb(*cbargs)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1493, in on_readable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     self.drain_events(connection)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1647, in drain_events
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     receiver = self.session.next_receiver(timeout=timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "<string>", line 6, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 777, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     if self._ecwait(lambda: self.incoming, timeout):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 57, in _ecwait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     result = self._ewait(lambda: self.closed or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     self.check_error()
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     raise e
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984) ConnectionError: [Errno 101] Network is unreachable

Actions

Copy link

Updated by daviddavis over 6 years ago

I'm pretty sure that this is the culprit here: https://git.io/vDABe. From my testing it looks like localhost resolves to ::1 before those lines get called:

$ telnet localhost 5671
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Then after:

$ telnet localhost 5671
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

I tried testing this out by removing this line from /etc/hosts on the box after it came up but before the BATS tests get run:

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

After that, I the BATS tests passed (three out of three times I tested) which seems to confirm my suspicion. I think Pulp is connected or trying to connect to qpid via ::1 as ipv6 gets disabled.

I see three possible solutions:

1. Restart pulp after disabling ipv6 in BATS
2. Have ansible or BATS update /etc/hosts to remove the ::1 line
3. Configure pulp to point to qpid at 127.0.0.1 instead of localhost

Option 1 may not work as long as localhost could resolve to ::1. Option 3 is a little hard because the certs point to localhost.

Regardless though, I think this is probably a Katello bug. We could maybe make it clearer in the pulp logs when it can't connect to qpid but not sure how helpful that would be over a "Network unreachable" error like we do now.

Actions

Copy link