Pulp

Could we get some logging related to these failures?

http://docs.pulpproject.org/en/2.8/user-guide/troubleshooting.html

Sorry,

Will do

In looking at the logs, Qpid is raising the following connection error:

beat raised exception <class 'qpid.messaging.exceptions.Timeout'>: Timeout('Connection attach timed out',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler
    beat.start()
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
    interval = self.scheduler.tick()
  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 265, in tick
    ret = self.call_tick(self, celerybeat_name)
  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 230, in call_tick
    ret = super(Scheduler, self).tick()
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick 
    next_time_to_run = self.maybe_due(entry, self.publisher)
  File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
    return self.Publisher(self._ensure_connected())
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in _ensure_connected
    _error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
    interval_start, interval_step, interval_max, callback)
  File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
    return self.connection
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
    self._qpid_conn = establish(**self.connection_options)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
    conn.open(timeout=timeout)
  File "<string>", line 6, in open
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open 
    self.attach(timeout=timeout)
  File "<string>", line 6, in attach
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 343, in attach
    raise Timeout("Connection attach timed out")
Timeout: Connection attach timed out

It has the same symptom as this BZ but a different root cause. https://bugzilla.redhat.com/show_bug.cgi?id=1096289

This looks like a Qpid issue. How reproducable is this on the system you experience the problem?

[its pretty common that when i run a katello-service restart or stop the services and start them back they beat service will fail

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.