Project

Profile

Help

Issue #431

closed

A key-error for connection_errors Stack dump appears when stopping pulp workers from time to time

Added by mkovacik@redhat.com about 9 years ago. Updated about 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Low
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.4 Beta
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem:
The stack dump from "Additional Info" can be seen in logs when starting/stopping pulp workers

Version-Release number of selected component (if applicable):
pulp-server-2.4.0-0.14.beta.fc20.noarch
python-kombu-3.0.15-5.pulp.fc20.noarch

How reproducible:
~1/2

Steps to Reproduce:
1. systemctl stop `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1
2. check /var/log messages for the stack dump
2. ystemctl start `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1
4. goto 1.

Actual results:
repeating few times, the stack dump from additional info appears

Expected results:
no stack dump in during stop workers appears when accessing the connection_errors property

Additional info:

  1. the stack dump
    May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping The Apache HTTP Server...
    May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp's Celerybeat...
    May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp Resource Manager...
    May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp Worker #0...
    May 16 13:37:10 ec2-54-220-31-64 systemd: Stopping Pulp Celery Workers...
    May 16 13:37:10 ec2-54-220-31-64 celery: celery beat v3.1.11 (Cipater) is starting.
    May 16 13:37:10 ec2-54-220-31-64 celery: _ - ... _ -
    May 16 13:37:10 ec2-54-220-31-64 celery: Configuration ->
    May 16 13:37:10 ec2-54-220-31-64 celery: . broker -> qpid://:5672//
    May 16 13:37:10 ec2-54-220-31-64 celery: . loader -> celery.loaders.app.AppLoader
    May 16 13:37:10 ec2-54-220-31-64 celery: . scheduler -> pulp.server.async.scheduler.Scheduler
    May 16 13:37:10 ec2-54-220-31-64 celery: . logfile -> [stderr]@%INFO
    May 16 13:37:10 ec2-54-220-31-64 celery: . maxinterval -> now (0s)
    May 16 13:37:10 ec2-54-220-31-64 celery: worker: Warm shutdown (MainProcess)
    May 16 13:37:10 ec2-54-220-31-64 celery: worker: Warm shutdown (MainProcess)
    May 16 13:37:10 ec2-54-220-31-64 celery: Exception in thread Thread-2 (most likely raised during interpreter shutdown):
    May 16 13:37:10 ec2-54-220-31-64 celery: Traceback (most recent call last):
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib64/python2.7/threading.py", line 811, inbootstrap_inner
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib64/python2.7/threading.py", line 764, in run
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 92, in monitor_events
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 732, in __exit
    _
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 334, in release
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 302, in _close
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 292, in _do_close_self
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 285, in maybe_close_channel
    May 16 13:37:10 ec2-54-220-31-64 celery: File "/usr/lib/python2.7/site-packages/kombu/utils/_init__.py", line 320, inget_
    May 16 13:37:10 ec2-54-220-31-64 celery: <type 'exceptions.KeyError'>: 'connection_errors'
    May 16 13:37:10 ec2-54-220-31-64 systemd: Stopped Pulp's Celerybeat.
    May 16 13:37:11 ec2-54-220-31-64 celery: ready.
    May 16 13:37:11 ec2-54-220-31-64 celery: ready.
    May 16 13:37:11 ec2-54-220-31-64 systemd: Stopped The Apache HTTP Server.
    May 16 13:37:12 ec2-54-220-31-64 systemd: Stopped Pulp Worker #0.

+ This bug was cloned from Bugzilla Bug #1098536 +

Also available in: Atom PDF