Project

Profile

Help

Issue #2664

Updated by bmbouter over 4 years ago

To reproduce:

Create a 'zoo' repo that syncs from the zoo fixture repo

<code>pulp-admin rpm repo create --repo-id tester --feed https://repos.fedorapeople.org/repos/pulp/pulp/fixtures/rpm/</code>

Sync 'zoo' manually once to see it working. Sync it again, and just when the task begins run:

<code>sudo pkill -9 -f reserved_resource_worker</code>

Then stop qpidd with <code>sudo systemctl stop qpidd</code>

Then restart the pulp_workers <code>sudo systemctl restart pulp_workers</code>

Observe the traceback:

<pre>
Mar 24 17:25:50 dev pulp[10046]: pulp.server.async.tasks:INFO: Cleaning up shutdown worker 'reserved_resource_worker-0@dev'.
Mar 24 17:25:50 dev celery[10046]: Traceback (most recent call last):
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib64/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
Mar 24 17:25:50 dev celery[10046]: finalizer()
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib64/python2.7/multiprocessing/util.py", line 207, in __call__
Mar 24 17:25:50 dev celery[10046]: res = self._callback(*self._args, **self._kwargs)
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 201, in _send_worker_shutdown
Mar 24 17:25:50 dev celery[10046]: signals.worker_shutdown.send(sender=self)
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 166, in send
Mar 24 17:25:50 dev celery[10046]: response = receiver(signal=self, sender=sender, **named)
Mar 24 17:25:50 dev celery[10046]: File "/home/vagrant/devel/pulp/server/pulp/server/async/app.py", line 194, in shutdown_worker
Mar 24 17:25:50 dev celery[10046]: tasks._delete_worker(sender.hostname, normal_shutdown=True)
Mar 24 17:25:50 dev celery[10046]: File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 273, in _delete_worker
Mar 24 17:25:50 dev celery[10046]: cancel(task_status['task_id'])
Mar 24 17:25:50 dev celery[10046]: File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 642, in cancel
Mar 24 17:25:50 dev celery[10046]: controller.revoke(task_id, terminate=True)
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/celery/app/control.py", line 172, in revoke
Mar 24 17:25:50 dev celery[10046]: 'signal': signal}, **kwargs)
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/celery/app/control.py", line 316, in broadcast
Mar 24 17:25:50 dev celery[10046]: limit, callback, channel=channel,
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/kombu/pidbox.py", line 283, in _broadcast
Mar 24 17:25:50 dev celery[10046]: chan = channel or self.connection.default_channel
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel
Mar 24 17:25:50 dev celery[10046]: self.connection
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
Mar 24 17:25:50 dev celery[10046]: self._connection = self._establish_connection()
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
Mar 24 17:25:50 dev celery[10046]: conn = self.transport.establish_connection()
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
Mar 24 17:25:50 dev celery[10046]: conn = self.Connection(**opts)
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
Mar 24 17:25:50 dev celery[10046]: self._qpid_conn = establish(**self.connection_options)
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
Mar 24 17:25:50 dev celery[10046]: conn.open(timeout=timeout)
Mar 24 17:25:50 dev celery[10046]: File "<string>", line 6, in open
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
Mar 24 17:25:50 dev celery[10046]: self.attach(timeout=timeout)
Mar 24 17:25:50 dev celery[10046]: File "<string>", line 6, in attach
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 341, in attach
Mar 24 17:25:50 dev celery[10046]: if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Mar 24 17:25:50 dev celery[10046]: self.check_error()
Mar 24 17:25:50 dev celery[10046]: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Mar 24 17:25:50 dev celery[10046]: raise e
Mar 24 17:25:50 dev celery[10046]: ConnectError: [Errno 111] Connection refused
</pre>

Then start qpidd again <code>sudo systemctl start qpidd</code>

Note Trigger a new sync and observe that the pulp_workers never started. They emit the traceback and shut back down. it does not complete also. The workers are in a dirty state.

Back