Project

Profile

Help

Issue #2173

/bin/systemctl status pulp_celerybeat.service fails to start and will fail until several attempts have been tried

Added by ricklang2010@gmail.com over 5 years ago. Updated over 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.4
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Looks as if this old bugzilla.redhat bug (1096289
)has showed back up in at least on certain hardware.

I get this trouble with a fresh install of centos7.1511 and katello 3.0 installation.

The following packages are installed supporting this installation

candlepin-0.9.54.6-1.el7.noarch
candlepin-selinux-0.9.54.6-1.el7.noarch
dscaprv01.corp.acxiom.net-qpid-broker-1.0-1.noarch
dscaprv01.corp.acxiom.net-qpid-client-cert-1.0-1.noarch
dscaprv01.corp.acxiom.net-qpid-router-client-1.0-1.noarch
dscaprv01.corp.acxiom.net-qpid-router-server-1.0-1.noarch
foreman-installer-katello-3.0.2-1.el7.noarch
katello-3.0.0-9.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-client-bootstrap-1.1.0-1.el7.noarch
katello-common-3.0.0-9.el7.noarch
katello-debug-3.0.0-9.el7.noarch
katello-default-ca-1.0-1.noarch
katello-installer-base-3.0.2-1.el7.noarch
katello-repos-3.0.0-3.el7.noarch
katello-selinux-3.0.1-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.0.0-9.el7.noarch
libqpid-dispatch-0.4-11.el7.x86_64
pulp-client-1.0-1.noarch
pulp-docker-plugins-2.0.1-1.el7.noarch
pulp-katello-1.0-1.el7.noarch
pulp-puppet-plugins-2.8.4-1.el7.noarch
pulp-puppet-tools-2.8.4-1.el7.noarch
pulp-rpm-plugins-2.8.4-1.el7.noarch
pulp-selinux-2.8.4-1.el7.noarch
pulp-server-2.8.4-1.el7.noarch
python-gofer-qpid-2.7.6-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-5.pulp.el7.noarch
python-pulp-common-2.8.4-1.el7.noarch
python-pulp-docker-common-2.0.1-1.el7.noarch
python-pulp-oid_validation-2.8.4-1.el7.noarch
python-pulp-puppet-common-2.8.4-1.el7.noarch
python-pulp-repoauth-2.8.4-1.el7.noarch
python-pulp-rpm-common-2.8.4-1.el7.noarch
python-pulp-streamer-2.8.4-1.el7.noarch
python-qpid-0.30-9.el7.noarch
python-qpid-qmf-0.30-5.el7.x86_64
qpid-cpp-client-0.30-8.el7.x86_64
qpid-cpp-client-devel-0.30-8.el7.x86_64
qpid-cpp-server-0.30-8.el7.x86_64
qpid-cpp-server-linearstore-0.30-8.el7.x86_64
qpid-dispatch-router-0.4-11.el7.x86_64
qpid-proton-c-0.9-13.el7.x86_64
qpid-qmf-0.30-5.el7.x86_64
qpid-tools-0.30-4.el7.noarch
rubygem-smart_proxy_pulp-1.2.2-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.0.25-1.el7.noarch
tfm-rubygem-katello-3.0.2-1.el7.noarch
tfm-rubygem-qpid_messaging-0.30.0-7.el7.x86_64

running on a dell 720.

Note if I install this product in a vm it works as expected

captured_celery_sani.tar (1.21 MB) captured_celery_sani.tar ricklang2010@gmail.com, 08/19/2016 04:37 PM

History

#1 Updated by semyers over 5 years ago

Could we get some logging related to these failures?

http://docs.pulpproject.org/en/2.8/user-guide/troubleshooting.html

#2 Updated by ricklang2010@gmail.com over 5 years ago

Sorry,

Will do

#3 Updated by ricklang2010@gmail.com over 5 years ago

I captured a few snippets that may shine some light. Let me know if i need to gather more data. Thank you

#4 Updated by bmbouter over 5 years ago

In looking at the logs, Qpid is raising the following connection error:

beat raised exception <class 'qpid.messaging.exceptions.Timeout'>: Timeout('Connection attach timed out',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler
    beat.start()
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
    interval = self.scheduler.tick()
  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 265, in tick
    ret = self.call_tick(self, celerybeat_name)
  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 230, in call_tick
    ret = super(Scheduler, self).tick()
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick 
    next_time_to_run = self.maybe_due(entry, self.publisher)
  File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
    return self.Publisher(self._ensure_connected())
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in _ensure_connected
    _error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
    interval_start, interval_step, interval_max, callback)
  File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
    return self.connection
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
    self._qpid_conn = establish(**self.connection_options)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
    conn.open(timeout=timeout)
  File "<string>", line 6, in open
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open 
    self.attach(timeout=timeout)
  File "<string>", line 6, in attach
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 343, in attach
    raise Timeout("Connection attach timed out")
Timeout: Connection attach timed out

It has the same symptom as this BZ but a different root cause. https://bugzilla.redhat.com/show_bug.cgi?id=1096289

This looks like a Qpid issue. How reproducable is this on the system you experience the problem?

#5 Updated by amacdona@redhat.com over 5 years ago

  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#6 Updated by ricklang2010@gmail.com over 5 years ago

[its pretty common that when i run a katello-service restart or stop the services and start them back they beat service will fail

#7 Updated by bmbouter over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

#8 Updated by bmbouter over 2 years ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#9 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF