Issue #2173
closed/bin/systemctl status pulp_celerybeat.service fails to start and will fail until several attempts have been tried
Description
Looks as if this old bugzilla.redhat bug (1096289
)has showed back up in at least on certain hardware.
I get this trouble with a fresh install of centos7.1511 and katello 3.0 installation.
The following packages are installed supporting this installation
candlepin-0.9.54.6-1.el7.noarch
candlepin-selinux-0.9.54.6-1.el7.noarch
dscaprv01.corp.acxiom.net-qpid-broker-1.0-1.noarch
dscaprv01.corp.acxiom.net-qpid-client-cert-1.0-1.noarch
dscaprv01.corp.acxiom.net-qpid-router-client-1.0-1.noarch
dscaprv01.corp.acxiom.net-qpid-router-server-1.0-1.noarch
foreman-installer-katello-3.0.2-1.el7.noarch
katello-3.0.0-9.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-client-bootstrap-1.1.0-1.el7.noarch
katello-common-3.0.0-9.el7.noarch
katello-debug-3.0.0-9.el7.noarch
katello-default-ca-1.0-1.noarch
katello-installer-base-3.0.2-1.el7.noarch
katello-repos-3.0.0-3.el7.noarch
katello-selinux-3.0.1-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.0.0-9.el7.noarch
libqpid-dispatch-0.4-11.el7.x86_64
pulp-client-1.0-1.noarch
pulp-docker-plugins-2.0.1-1.el7.noarch
pulp-katello-1.0-1.el7.noarch
pulp-puppet-plugins-2.8.4-1.el7.noarch
pulp-puppet-tools-2.8.4-1.el7.noarch
pulp-rpm-plugins-2.8.4-1.el7.noarch
pulp-selinux-2.8.4-1.el7.noarch
pulp-server-2.8.4-1.el7.noarch
python-gofer-qpid-2.7.6-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-5.pulp.el7.noarch
python-pulp-common-2.8.4-1.el7.noarch
python-pulp-docker-common-2.0.1-1.el7.noarch
python-pulp-oid_validation-2.8.4-1.el7.noarch
python-pulp-puppet-common-2.8.4-1.el7.noarch
python-pulp-repoauth-2.8.4-1.el7.noarch
python-pulp-rpm-common-2.8.4-1.el7.noarch
python-pulp-streamer-2.8.4-1.el7.noarch
python-qpid-0.30-9.el7.noarch
python-qpid-qmf-0.30-5.el7.x86_64
qpid-cpp-client-0.30-8.el7.x86_64
qpid-cpp-client-devel-0.30-8.el7.x86_64
qpid-cpp-server-0.30-8.el7.x86_64
qpid-cpp-server-linearstore-0.30-8.el7.x86_64
qpid-dispatch-router-0.4-11.el7.x86_64
qpid-proton-c-0.9-13.el7.x86_64
qpid-qmf-0.30-5.el7.x86_64
qpid-tools-0.30-4.el7.noarch
rubygem-smart_proxy_pulp-1.2.2-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.0.25-1.el7.noarch
tfm-rubygem-katello-3.0.2-1.el7.noarch
tfm-rubygem-qpid_messaging-0.30.0-7.el7.x86_64
running on a dell 720.
Note if I install this product in a vm it works as expected
Files
Updated by semyers over 7 years ago
Could we get some logging related to these failures?
http://docs.pulpproject.org/en/2.8/user-guide/troubleshooting.html
Updated by ricklang2010@gmail.com over 7 years ago
- File captured_celery_sani.tar captured_celery_sani.tar added
I captured a few snippets that may shine some light. Let me know if i need to gather more data. Thank you
Updated by bmbouter over 7 years ago
In looking at the logs, Qpid is raising the following connection error:
beat raised exception <class 'qpid.messaging.exceptions.Timeout'>: Timeout('Connection attach timed out',)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler
beat.start()
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
interval = self.scheduler.tick()
File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 265, in tick
ret = self.call_tick(self, celerybeat_name)
File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 230, in call_tick
ret = super(Scheduler, self).tick()
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
next_time_to_run = self.maybe_due(entry, self.publisher)
File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
return self.Publisher(self._ensure_connected())
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in _ensure_connected
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
interval_start, interval_step, interval_max, callback)
File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
return fun(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
return self.connection
File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
self._connection = self._establish_connection()
File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
conn = self.Connection(**opts)
File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
self._qpid_conn = establish(**self.connection_options)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
conn.open(timeout=timeout)
File "<string>", line 6, in open
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
self.attach(timeout=timeout)
File "<string>", line 6, in attach
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 343, in attach
raise Timeout("Connection attach timed out")
Timeout: Connection attach timed out
It has the same symptom as this BZ but a different root cause. https://bugzilla.redhat.com/show_bug.cgi?id=1096289
This looks like a Qpid issue. How reproducable is this on the system you experience the problem?
Updated by amacdona@redhat.com over 7 years ago
- Severity changed from 2. Medium to 3. High
- Triaged changed from No to Yes
Updated by ricklang2010@gmail.com over 7 years ago
[its pretty common that when i run a katello-service restart or stop the services and start them back they beat service will fail
Updated by bmbouter about 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter about 5 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.