Actions
Issue #2611
closed
task system crashes with qpid + ssl unless client certs are provided
Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.10.3
Platform Release:
OS:
Fedora 25
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
This may be a bug in kombu, but I see the behavior in pulp.
Given these settings in /etc/pulp/server.conf:
[tasks]
broker_url: qpid://qpidd:5671
celery_require_ssl: true
cacert: /var/run/secrets/pulp/pulp-ca/ca.crt
Pulp assumes that a client certificate will be present. If you only want to enjoy a TLS-enabled connection, without using client certificates, that appears to be impossible. The worker exits after it fails to open the client certificate file at the default location.
As a side note, the error message is unhelpful. Nothing tells the user which file failed to open. See the traceback below.
Using strace I was able to see which file it was failing to open:
[pid 59] open("/etc/pki/pulp/qpid/client.crt", O_RDONLY) = -1 ENOENT (No such file or directory)
I looked for the log file passed to the worker using the "--logfile" option at startup, but the file did not exist.
This is the traceback seen in the system log:
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) Unrecoverable error: InternalError('Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch\n self.connect()\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect\n self._transport = trans(self.connection, host, port)\n File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__\n cert_reqs=validate)\n File "/usr/lib64/python2.7/ssl.py", line 943, in wrap_socket\n ciphers=ciphers)\n File "/usr/lib64/python2.7/ssl.py", line 554, in __init__\n self._context.load_cert_chain(certfile, keyfile)\nIOError: [Errno 2] No such file or directory\n',)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) Traceback (most recent call last):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.blueprint.start(self)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) step.start(parent)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) return self.obj.start()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) blueprint.start(self)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) step.start(parent)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 479, in start
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) c.connection = c.connect()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 376, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) callback=maybe_shutdown,
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) interval_start, interval_step, interval_max, callback)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) return fun(*args, **kwargs)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) return self.connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._connection = self._establish_connection()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) conn = self.transport.establish_connection()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) conn = self.Connection(**opts)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._qpid_conn = establish(**self.connection_options)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) conn.open(timeout=timeout)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "<string>", line 6, in open
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.attach(timeout=timeout)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "<string>", line 6, in attach
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 341, in attach
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.check_error()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) raise e
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) InternalError: Traceback (most recent call last):
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self.connect()
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._transport = trans(self.connection, host, port)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) cert_reqs=validate)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib64/python2.7/ssl.py", line 943, in wrap_socket
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) ciphers=ciphers)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) File "/usr/lib64/python2.7/ssl.py", line 554, in __init__
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) self._context.load_cert_chain(certfile, keyfile)
Feb 26 20:58:10 minikube pulp[16860]: celery.worker:ERROR: (59-16832) IOError: [Errno 2] No such file or directory
After providing a valid client certificate and key, the worker was able to start normally and connect to the broker.
python-qpid-common-1.35.0-1.fc25.noarch
qpid-proton-c-0.14.0-1.fc25.x86_64
python-gofer-qpid-2.9.2-2.fc25.noarch
python-qpid-1.35.0-1.fc25.noarch
qpid-cpp-client-1.35.0-1.fc25.x86_64
qpid-qmf-1.35.0-1.fc25.x86_64
qpid-tools-1.35.0-1.fc25.noarch
python-qpid-qmf-1.35.0-1.fc25.x86_64
python-kombu-3.0.33-7.fc25.noarch
python2-celery-3.1.20-3.fc25.noarch
Updated by bmbouter about 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter about 5 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
Actions
.