Project

Profile

Help

Issue #2586

Pulp workers/beat/resource_manager go missing

Added by ehelms@redhat.com over 3 years ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
Urgent
Assignee:
Category:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 15
Quarter:

Description

Running on Pulp 2.10.3, I see the following error messages running this setup:

clone https://github.com/Katello/forklift
vagrant up centos7-katello-nightly

If you do not see the error messages initially:

vagrant ssh centos7-katello-nightly
cd /
/vagrant/forklift/bats/bootstrap_vagrant.sh

cd /vagrant/forklift/bats
bats fb-install-katello.bats fb-content-katello.bats
Feb 13 20:45:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:46:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:47:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:48:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:49:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Feb 13 20:50:22 centos7 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.

History

#1 Updated by bizhang over 3 years ago

  • Priority changed from Normal to Urgent
  • Sprint/Milestone set to 33
  • Triaged changed from No to Yes

#2 Updated by daviddavis over 3 years ago

  • Assignee set to daviddavis

#3 Updated by daviddavis over 3 years ago

  • Status changed from NEW to ASSIGNED

#4 Updated by daviddavis over 3 years ago

It looks like pulp is having problems connecting to qpid:

Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136) consumer: Connection to broker lost. Trying to re-establish the connection...
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136) Traceback (most recent call last):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     blueprint.start(self)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     step.start(parent)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376) consumer: Connection to broker lost. Trying to re-establish the connection...
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     c.loop(*c.loop_args())
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376) Traceback (most recent call last):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     next(loop)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     blueprint.start(self)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     cb(*cbargs)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     step.start(parent)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1493, in on_readable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     self.drain_events(connection)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1647, in drain_events
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     c.loop(*c.loop_args())
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     receiver = self.session.next_receiver(timeout=timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "<string>", line 6, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     next(loop)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 777, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     cb(*cbargs)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     if self._ecwait(lambda: self.incoming, timeout):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 57, in _ecwait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1493, in on_readable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     result = self._ewait(lambda: self.closed or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     self.drain_events(connection)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1647, in drain_events
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     receiver = self.session.next_receiver(timeout=timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "<string>", line 6, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     self.check_error()
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 777, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     if self._ecwait(lambda: self.incoming, timeout):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136)     raise e
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 57, in _ecwait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (907-31136) ConnectionError: [Errno 101] Network is unreachable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     result = self._ewait(lambda: self.closed or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     self.check_error()
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376)     raise e
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (939-01376) ConnectionError: [Errno 101] Network is unreachable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984) consumer: Connection to broker lost. Trying to re-establish the connection...
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984) Traceback (most recent call last):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     blueprint.start(self)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     step.start(parent)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     c.loop(*c.loop_args())
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     next(loop)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     cb(*cbargs)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1493, in on_readable
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     self.drain_events(connection)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1647, in drain_events
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     receiver = self.session.next_receiver(timeout=timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "<string>", line 6, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 777, in next_receiver
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     if self._ecwait(lambda: self.incoming, timeout):
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 57, in _ecwait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     result = self._ewait(lambda: self.closed or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     self.check_error()
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984)     raise e
Feb 17 19:25:11 pipeline-katello-nightly-centos7 pulp: celery.worker.consumer:WARNING: (937-01984) ConnectionError: [Errno 101] Network is unreachable

#5 Updated by daviddavis over 3 years ago

I'm pretty sure that this is the culprit here: https://git.io/vDABe. From my testing it looks like localhost resolves to ::1 before those lines get called:

$ telnet localhost 5671
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Then after:

$ telnet localhost 5671
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

I tried testing this out by removing this line from /etc/hosts on the box after it came up but before the BATS tests get run:

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

After that, I the BATS tests passed (three out of three times I tested) which seems to confirm my suspicion. I think Pulp is connected or trying to connect to qpid via ::1 as ipv6 gets disabled.

I see three possible solutions:

1. Restart pulp after disabling ipv6 in BATS
2. Have ansible or BATS update /etc/hosts to remove the ::1 line
3. Configure pulp to point to qpid at 127.0.0.1 instead of localhost

Option 1 may not work as long as localhost could resolve to ::1. Option 3 is a little hard because the certs point to localhost.

Regardless though, I think this is probably a Katello bug. We could maybe make it clearer in the pulp logs when it can't connect to qpid but not sure how helpful that would be over a "Network unreachable" error like we do now.

#6 Updated by daviddavis over 3 years ago

  • Status changed from ASSIGNED to CLOSED - NOTABUG

#7 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 15

#8 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (33)

#9 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF