Issue #431
closedA key-error for connection_errors Stack dump appears when stopping pulp workers from time to time
Description
Description of problem:
The stack dump from "Additional Info" can be seen in logs when starting/stopping pulp workers
Version-Release number of selected component (if applicable):
pulp-server-2.4.0-0.14.beta.fc20.noarch
python-kombu-3.0.15-5.pulp.fc20.noarch
How reproducible:
~1/2
Steps to Reproduce:
1. systemctl stop `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1
2. check /var/log messages for the stack dump
2. ystemctl start `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1
4. goto 1.
Actual results:
repeating few times, the stack dump from additional info appears
Expected results:
no stack dump in during stop workers appears when accessing the connection_errors property
Additional info:
- the stack dump
May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping The Apache HTTP Server...
May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp's Celerybeat...
May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp Resource Manager...
May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp Worker #0...
May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp Celery Workers...
May 16 13:37:10 ec2-54-220-31-64 celery: celery beat v3.1.11 (Cipater) is starting.
May 16 13:37:10 ec2-54-220-31-64 celery: _ - ... _ -
May 16 13:37:10 ec2-54-220-31-64 celery: Configuration ->
May 16 13:37:10 ec2-54-220-31-64 celery: . broker -> qpid://guest@ec2-54-220-31-64.eu-west-1.compute.amazonaws.com:5672//
May 16 13:37:10 ec2-54-220-31-64 celery: . loader -> celery.loaders.app.AppLoader
May 16 13:37:10 ec2-54-220-31-64 celery: . scheduler -> pulp.server.async.scheduler.Scheduler
May 16 13:37:10 ec2-54-220-31-64 celery: . logfile -> [stderr]@%INFO
May 16 13:37:10 ec2-54-220-31-64 celery: . maxinterval -> now (0s)
May 16 13:37:10 ec2-54-220-31-64 celery: worker: Warm shutdown (MainProcess)
May 16 13:37:10 ec2-54-220-31-64 celery: worker: Warm shutdown (MainProcess)
May 16 13:37:10 ec2-54-220-31-64 celery: Exception in thread Thread-2 (most likely raised during interpreter shutdown):
May 16 13:37:10 ec2-54-220-31-64 celery: Traceback (most recent call last):
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib64/python2.7/threading.py", line 811, inbootstrap_inner
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib64/python2.7/threading.py", line 764, in run
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 92, in monitor_events
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 732, in __exit_
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 334, in release
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 302, in _close
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 292, in _do_close_self
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 285, in maybe_close_channel
May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/utils/_init__.py", line 320, inget_
May 16 13:37:10 ec2-54-220-31-64 celery: <type 'exceptions.KeyError'>: 'connection_errors'
May 16 13:37:10 ec2-54-220-31-64 systemd: Stopped Pulp's Celerybeat.
May 16 13:37:11 ec2-54-220-31-64 celery: reserved_resource_worker-0@ec2-54-220-31-64.eu-west-1.compute.amazonaws.com ready.
May 16 13:37:11 ec2-54-220-31-64 celery: resource_manager@ec2-54-220-31-64.eu-west-1.compute.amazonaws.com ready.
May 16 13:37:11 ec2-54-220-31-64 systemd: Stopped The Apache HTTP Server.
May 16 13:37:12 ec2-54-220-31-64 systemd: Stopped Pulp Worker #0.
+ This bug was cloned from Bugzilla Bug #1098536 +