Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2611

closed

task system crashes with qpid + ssl unless client certs are provided

Added by mhrivnak about 7 years ago. Updated about 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.10.3
Platform Release:
OS:
Fedora 25
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

This may be a bug in kombu, but I see the behavior in pulp.

Given these settings in /etc/pulp/server.conf:

[tasks]
broker_url: qpid://qpidd:5671
celery_require_ssl: true
cacert: /var/run/secrets/pulp/pulp-ca/ca.crt

Pulp assumes that a client certificate will be present. If you only want to enjoy a TLS-enabled connection, without using client certificates, that appears to be impossible. The worker exits after it fails to open the client certificate file at the default location.

As a side note, the error message is unhelpful. Nothing tells the user which file failed to open. See the traceback below.

Using strace I was able to see which file it was failing to open:

[pid    59] open("/etc/pki/pulp/qpid/client.crt", O_RDONLY) = -1 ENOENT (No such file or directory)

I looked for the log file passed to the worker using the "--logfile" option at startup, but the file did not exist.

This is the traceback seen in the system log:

Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) Unrecoverable error: InternalError('Traceback (most recent call last):\n  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch\n    self.connect()\n  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect\n    self._transport = trans(self.connection, host, port)\n  File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__\n    cert_reqs=validate)\n  File "/usr/lib64/python2.7/ssl.py", line 943, in wrap_socket\n    ciphers=ciphers)\n  File "/usr/lib64/python2.7/ssl.py", line 554, in __init__\n    self._context.load_cert_chain(certfile, keyfile)\nIOError: [Errno 2] No such file or directory\n',)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) Traceback (most recent call last):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self.blueprint.start(self)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     step.start(parent)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     return self.obj.start()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     blueprint.start(self)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     step.start(parent)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 479, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     c.connection = c.connect()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 376, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     callback=maybe_shutdown,
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     interval_start, interval_step, interval_max, callback)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     return fun(*args, **kwargs)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     return self.connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self._connection = self._establish_connection()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     conn = self.transport.establish_connection()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     conn = self.Connection(**opts)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self._qpid_conn = establish(**self.connection_options)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     conn.open(timeout=timeout)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "<string>", line 6, in open
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self.attach(timeout=timeout)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "<string>", line 6, in attach
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 341, in attach
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self.check_error()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     raise e
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) InternalError: Traceback (most recent call last):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self.connect()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self._transport = trans(self.connection, host, port)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     cert_reqs=validate)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib64/python2.7/ssl.py", line 943, in wrap_socket
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     ciphers=ciphers)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)   File "/usr/lib64/python2.7/ssl.py", line 554, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832)     self._context.load_cert_chain(certfile, keyfile)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) IOError: [Errno 2] No such file or directory

After providing a valid client certificate and key, the worker was able to start normally and connect to the broker.

python-qpid-common-1.35.0-1.fc25.noarch
qpid-proton-c-0.14.0-1.fc25.x86_64
python-gofer-qpid-2.9.2-2.fc25.noarch
python-qpid-1.35.0-1.fc25.noarch
qpid-cpp-client-1.35.0-1.fc25.x86_64
qpid-qmf-1.35.0-1.fc25.x86_64
qpid-tools-1.35.0-1.fc25.noarch
python-qpid-qmf-1.35.0-1.fc25.x86_64
python-kombu-3.0.33-7.fc25.noarch
python2-celery-3.1.20-3.fc25.noarch

Actions
Send by e-mail Copy link

Also available in: Atom PDF