Issue #2611
closedtask system crashes with qpid + ssl unless client certs are provided
Description
This may be a bug in kombu, but I see the behavior in pulp.
Given these settings in /etc/pulp/server.conf:
[tasks]
broker_url: qpid://qpidd:5671
celery_require_ssl: true
cacert: /var/run/secrets/pulp/pulp-ca/ca.crt
Pulp assumes that a client certificate will be present. If you only want to enjoy a TLS-enabled connection, without using client certificates, that appears to be impossible. The worker exits after it fails to open the client certificate file at the default location.
As a side note, the error message is unhelpful. Nothing tells the user which file failed to open. See the traceback below.
Using strace I was able to see which file it was failing to open:
[pid 59] open("/etc/pki/pulp/qpid/client.crt", O_RDONLY) = -1 ENOENT (No such file or directory)
I looked for the log file passed to the worker using the "--logfile" option at startup, but the file did not exist.
This is the traceback seen in the system log:
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) Unrecoverable error: InternalError('Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch\n self.connect()\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect\n self._transport = trans(self.connection, host, port)\n File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__\n cert_reqs=validate)\n File "/usr/lib64/python2.7/ssl.py", line 943, in wrap_socket\n ciphers=ciphers)\n File "/usr/lib64/python2.7/ssl.py", line 554, in __init__\n self._context.load_cert_chain(certfile, keyfile)\nIOError: [Errno 2] No such file or directory\n',)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) Traceback (most recent call last):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.blueprint.start(self)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) step.start(parent)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) return self.obj.start()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) blueprint.start(self)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) step.start(parent)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 479, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) c.connection = c.connect()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 376, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) callback=maybe_shutdown,
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) interval_start, interval_step, interval_max, callback)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) return fun(*args, **kwargs)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) return self.connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._connection = self._establish_connection()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) conn = self.transport.establish_connection()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) conn = self.Connection(**opts)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._qpid_conn = establish(**self.connection_options)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) conn.open(timeout=timeout)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "<string>", line 6, in open
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.attach(timeout=timeout)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "<string>", line 6, in attach
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 341, in attach
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.check_error()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) raise e
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) InternalError: Traceback (most recent call last):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.connect()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._transport = trans(self.connection, host, port)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) cert_reqs=validate)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib64/python2.7/ssl.py", line 943, in wrap_socket
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) ciphers=ciphers)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib64/python2.7/ssl.py", line 554, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._context.load_cert_chain(certfile, keyfile)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) IOError: [Errno 2] No such file or directory
After providing a valid client certificate and key, the worker was able to start normally and connect to the broker.
python-qpid-common-1.35.0-1.fc25.noarch
qpid-proton-c-0.14.0-1.fc25.x86_64
python-gofer-qpid-2.9.2-2.fc25.noarch
python-qpid-1.35.0-1.fc25.noarch
qpid-cpp-client-1.35.0-1.fc25.x86_64
qpid-qmf-1.35.0-1.fc25.x86_64
qpid-tools-1.35.0-1.fc25.noarch
python-qpid-qmf-1.35.0-1.fc25.x86_64
python-kombu-3.0.33-7.fc25.noarch
python2-celery-3.1.20-3.fc25.noarch