Issue #2008: qpidd exceeded 2000 connections limit

Issue #2008

<pre> 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208) Traceback (most recent call last): 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       R = retval = fun(*args, **kwargs) 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 473, in __call__ 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       return super(Task, self).__call__(*args, **kwargs) 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 103, in __call__ 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       return super(PulpTask, self).__call__(*args, **kwargs) 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       return self.run(*args, **kwargs) 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 1162, in download_deferred 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       download_step.start() 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 1358, in start 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       self.downloader.download(self.download_requests) 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 142, in download 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       worker_thread.start() 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     File "/usr/lib64/python2.6/threading.py", line 474, in start 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)       _start_new_thread(self.__bootstrap, ()) 
 Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208) error: can't start new thread 
 Jun 13 21:41:43 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:43 [Security] error Client max total connection count limit of 2000 exceeded by 'qpid.127.0.0.1:5672-127.0.0.1:47832', user: 'guest@QPID'. Connection refused 
 Jun 13 21:41:43 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:43 [Security] error Client max total connection count limit of 2000 exceeded by 'qpid.127.0.0.1:5672-127.0.0.1:47833', user: 'anonymous'. Connection refused 
 Jun 13 21:41:43 pulp-docker01 pulp: kombu.transport.qpid:ERROR: Unable to authenticate to qpid using the following mechanisms: ['PLAIN', 'ANONYMOUS'] 
 Jun 13 21:41:53 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:53 [System] error Connection qpid.127.0.0.1:5672-127.0.0.1:47832 No protocol received closing 
 Jun 13 21:41:53 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:53 [System] error Connection qpid.127.0.0.1:5672-127.0.0.1:47833 No protocol received closing 
 Jun 13 21:42:46 pulp-docker01 qpidd[26643]: 2016-06-13 21:42:46 [Security] error Client max total connection count limit of 2000 exceeded by 'qpid.127.0.0.1:5672-127.0.0.1:47842', user: 'guest@QPID'. Connection refused 
 </pre> 

 


 that ^ repeated many times and at one point qpidd probably crashed, because logs changed to: 

 <pre> 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016) pulp.task 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016) Traceback (most recent call last): 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     File "/usr/lib/python2.6/site-packages/gofer/messaging/consumer.py", line 68, in _open 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)       self._reader.open() 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     File "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)       return fn(*args, **keywords) 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     File "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/model.py", line 587, in open 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)       self._impl.open() 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     File "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)       raise NotFound(*e.args) 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016) NotFound: no such queue: pulp.task 
 Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4181-50016) pulp.task 
 </pre> 


 Installed packages: 

 <pre> 
 qpid-tools-0.30-4.el6.noarch 
 python-qpid-0.30-6.el6.noarch 
 qpid-qmf-0.30-5.el6.x86_64 
 qpid-cpp-server-0.30-6.el6.x86_64 
 python-qpid-qmf-0.30-5.el6.x86_64 
 python-gofer-qpid-2.6.8-1.el6.noarch 
 qpid-cpp-client-0.30-6.el6.x86_64 
 qpid-proton-c-0.7-5.el6.x86_64 


 pulp-nodes-admin-extensions-2.8.1.2-1.el6sat.noarch 
 python-pulp-client-lib-2.8.1.2-1.el6sat.noarch 
 pulp-server-2.8.1.2-1.el6sat.noarch 
 pulp-selinux-2.8.1.2-1.el6sat.noarch 
 python-pulp-cdn-distributor-common-2.8.0-8.el6eng.noarch 
 python-pulp-agent-lib-2.8.1.2-1.el6sat.noarch 
 pulp-docker-admin-extensions-2.0.0.2-1.el6sat.noarch 
 pulp-nodes-consumer-extensions-2.8.1.2-1.el6sat.noarch 
 python-pulp-oid_validation-2.8.1.2-1.el6sat.noarch 
 pulp-agent-2.8.1.2-1.el6sat.noarch 
 m2crypto-0.21.1.pulp-8.el6.x86_64 
 python-pulp-repoauth-2.8.1.2-1.el6sat.noarch 
 pulp-consumer-client-2.8.1.2-1.el6sat.noarch 
 pulp-docker-plugins-2.0.0.2-1.el6sat.noarch 
 python-pulp-streamer-2.8.1.2-1.el6sat.noarch 
 python-isodate-0.5.0-1.pulp.el6.noarch 
 mod_wsgi-3.4-1.pulp.el6.x86_64 
 python-pulp-bindings-2.8.1.2-1.el6sat.noarch 
 pulp-admin-client-2.8.1.2-1.el6sat.noarch 
 pulp-cdn-distributor-plugins-2.8.0-8.el6eng.noarch 
 python-kombu-3.0.24-5.pulp.el6ui.noarch 
 python-pulp-docker-common-2.0.0.2-1.el6sat.noarch 
 pulp-nodes-common-2.8.1.2-1.el6sat.noarch 
 pulp-nodes-child-2.8.1.2-1.el6sat.noarch 
 python-pulp-common-2.8.1.2-1.el6sat.noarch 

 python-celery-3.1.11-1.el6.noarch 
 </pre>
Back
Project

Profile

Help

Pulp

Issue #2008
