Project

Profile

Help

Issue #2008

closed

qpidd exceeded 2000 connections limit

Added by jluza almost 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - WORKSFORME
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.0
Platform Release:
OS:
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 7
Quarter:

Description

Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208) Traceback (most recent call last):
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     R = retval = fun(*args, **kwargs)
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     return super(Task, self).__call__(*args, **kwargs)
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     return super(PulpTask, self).__call__(*args, **kwargs)
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     return self.run(*args, **kwargs)
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 1162, in download_deferred
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     download_step.start()
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 1358, in start
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     self.downloader.download(self.download_requests)
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 142, in download
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     worker_thread.start()
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)   File "/usr/lib64/python2.6/threading.py", line 474, in start
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208)     _start_new_thread(self.__bootstrap, ())
Jun 13 21:37:57 pulp-docker01 pulp: celery.worker.job:ERROR: (4062-22208) error: can't start new thread
Jun 13 21:41:43 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:43 [Security] error Client max total connection count limit of 2000 exceeded by 'qpid.127.0.0.1:5672-127.0.0.1:47832', user: 'guest@QPID'. Connection refused
Jun 13 21:41:43 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:43 [Security] error Client max total connection count limit of 2000 exceeded by 'qpid.127.0.0.1:5672-127.0.0.1:47833', user: 'anonymous'. Connection refused
Jun 13 21:41:43 pulp-docker01 pulp: kombu.transport.qpid:ERROR: Unable to authenticate to qpid using the following mechanisms: ['PLAIN', 'ANONYMOUS']
Jun 13 21:41:53 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:53 [System] error Connection qpid.127.0.0.1:5672-127.0.0.1:47832 No protocol received closing
Jun 13 21:41:53 pulp-docker01 qpidd[26643]: 2016-06-13 21:41:53 [System] error Connection qpid.127.0.0.1:5672-127.0.0.1:47833 No protocol received closing
Jun 13 21:42:46 pulp-docker01 qpidd[26643]: 2016-06-13 21:42:46 [Security] error Client max total connection count limit of 2000 exceeded by 'qpid.127.0.0.1:5672-127.0.0.1:47842', user: 'guest@QPID'. Connection refused

that ^ repeated many times and at one point qpidd probably crashed, because logs changed to:

Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016) pulp.task
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016) Traceback (most recent call last):
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)   File "/usr/lib/python2.6/site-packages/gofer/messaging/consumer.py", line 68, in _open
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     self._reader.open()
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)   File "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     return fn(*args, **keywords)
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)   File "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     self._impl.open()
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)   File "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016)     raise NotFound(*e.args)
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4179-50016) NotFound: no such queue: pulp.task
Jun 14 03:25:06 pulp-docker01 pulp: gofer.messaging.consumer:ERROR: (4181-50016) pulp.task

Installed packages:

qpid-tools-0.30-4.el6.noarch
python-qpid-0.30-6.el6.noarch
qpid-qmf-0.30-5.el6.x86_64
qpid-cpp-server-0.30-6.el6.x86_64
python-qpid-qmf-0.30-5.el6.x86_64
python-gofer-qpid-2.6.8-1.el6.noarch
qpid-cpp-client-0.30-6.el6.x86_64
qpid-proton-c-0.7-5.el6.x86_64

pulp-nodes-admin-extensions-2.8.1.2-1.el6sat.noarch
python-pulp-client-lib-2.8.1.2-1.el6sat.noarch
pulp-server-2.8.1.2-1.el6sat.noarch
pulp-selinux-2.8.1.2-1.el6sat.noarch
python-pulp-cdn-distributor-common-2.8.0-8.el6eng.noarch
python-pulp-agent-lib-2.8.1.2-1.el6sat.noarch
pulp-docker-admin-extensions-2.0.0.2-1.el6sat.noarch
pulp-nodes-consumer-extensions-2.8.1.2-1.el6sat.noarch
python-pulp-oid_validation-2.8.1.2-1.el6sat.noarch
pulp-agent-2.8.1.2-1.el6sat.noarch
m2crypto-0.21.1.pulp-8.el6.x86_64
python-pulp-repoauth-2.8.1.2-1.el6sat.noarch
pulp-consumer-client-2.8.1.2-1.el6sat.noarch
pulp-docker-plugins-2.0.0.2-1.el6sat.noarch
python-pulp-streamer-2.8.1.2-1.el6sat.noarch
python-isodate-0.5.0-1.pulp.el6.noarch
mod_wsgi-3.4-1.pulp.el6.x86_64
python-pulp-bindings-2.8.1.2-1.el6sat.noarch
pulp-admin-client-2.8.1.2-1.el6sat.noarch
pulp-cdn-distributor-plugins-2.8.0-8.el6eng.noarch
python-kombu-3.0.24-5.pulp.el6ui.noarch
python-pulp-docker-common-2.0.0.2-1.el6sat.noarch
pulp-nodes-common-2.8.1.2-1.el6sat.noarch
pulp-nodes-child-2.8.1.2-1.el6sat.noarch
python-pulp-common-2.8.1.2-1.el6sat.noarch

python-celery-3.1.11-1.el6.noarch

Also available in: Atom PDF