Pulp

This looks like a known issue that has been fixed in goferd 2.7.1 which is already available in Fedora 23 and Rawhide. I can't find the BZ at the moment.

It goes like this:

1. Send request(s) to an agent.
2. Stop goferd (before it has completed them).
3. Unregister the consumer/capsule.
4. Start goferd.

On restart, goferd will restart the unfinished request(s) which are routed to the katello-agent plugin. However, since the consumer/capsule is no longer registered, it's URL for the broker is 'None'.

A workaround is to do:

1. Purge outstanding requests for the katelloagent (plugin).

  $ cd /var/lib/gofer/messaging/pending/katelloagent
  $ rm *

2. Restart goferd.

A few choices going forward:

1. Back port fix to goferd 2.6.x
2. Switch to goferd 2.7

Katello is fine upgrading to goferd 2.7.1, are builds available for el5,6,7 for that package?

Not yet available for EL5/6/7 but I can get the ball rolling.

Hi

I follow the instructions you provide
$ cd /var/lib/gofer/messaging/pending/katelloagent
$ rm *
$ restart goferd

And still having this issues

Jan 21 04:29:18 awscapsule01 goferd: [INFO][pulp.agent.c5ca0064-6e6f-4557-8535-5df729b07580] root:501 - connected to my server:5647
Jan 21 04:29:19 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: my server
Jan 21 04:29:19 awscapsule01 goferd: [INFO][worker-0] gofer.rmi.dispatcher:600 - call: Content.update() sn=bbfa9d76-7160-471c-b1cf-64b741cc1db6 data={'task_id': '774781db-982b-4a47-8a09-fb3ca0584ce1', 'consumer_id': 'c5ca0064-6e6f-4557-8535-5df729b07580'}
Jan 21 04:29:20 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:21 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:21 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:21 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:23 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:24 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:26 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:27 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:28 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:28 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:29 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:30 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:30 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:31 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:31 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:31 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(*args, **keywords)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 842, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self._impl.send(address, signed, ttl)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 44, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(messenger, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 64, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(sender, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/producer.py", line 99, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - sender = self.connection.sender(address)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 125, in sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return self._impl.create_sender(address, name=name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 203, in create_sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return BlockingSender(self, self.container.create_sender(self.conn, utf8(address), name=utf8(name), handler=handler, options=options))
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 67, ininit_
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - super(BlockingSender, self).__init__(connection, sender)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 36, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - msg="Opening link %s" % link.name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 231, in wait
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self.container.process()
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3729, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - ev.dispatch(self.handler)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3654, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - result = dispatch(handler, type.method, self)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3543, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return m(*args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 254, in on_connection_remote_close
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - raise ConnectionClosed(event.connection)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 178, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - txt = "Connection %s closed" % self.url
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - AttributeError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - send (progress), failed
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/agent/rmi.py", line 239, in report
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - details=self.details)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 44, in fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - raise ModelError(*e.args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - ModelError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(*args, **keywords)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 842, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self._impl.send(address, signed, ttl)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 44, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(messenger, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 64, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(sender, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/producer.py", line 99, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - sender = self.connection.sender(address)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 125, in sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return self._impl.create_sender(address, name=name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 203, in create_sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return BlockingSender(self, self.container.create_sender(self.conn, utf8(address), name=utf8(name), handler=handler, options=options))
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 67, ininit_
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - super(BlockingSender, self).__init__(connection, sender)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 36, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - msg="Opening link %s" % link.name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 231, in wait
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self.container.process()
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3729, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - ev.dispatch(self.handler)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3654, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - result = dispatch(handler, type.method, self)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3543, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return m(*args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 254, in on_connection_remote_close
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - raise ConnectionClosed(event.connection)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 178, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - txt = "Connection %s closed" % self.url
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - AttributeError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - send (progress), failed
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/agent/rmi.py", line 239, in report
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - details=self.details)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 44, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - raise ModelError(*e.args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - ModelError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [INFO][worker-0] pulp_node.handlers.model:475 - Importer: Centos-6/nodes_http_importer, updated
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[1b48a5b1-2d9c-41a1-8b60-16f5b3552148]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.importer.update_importer_config[305e3a0b-8f6f-4c19-87ce-1688735adf5a]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[6d55c27f-c57f-4f8f-a912-1fbff269fa7c]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[1b48a5b1-2d9c-41a1-8b60-16f5b3552148] succeeded in 0.0352381469997s: None
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[b99e1a06-dd70-46e1-9f24-d5f2ff694f7b]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[6913aa59-c51e-4924-bf1d-b6974aef83ce]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[a7d17f8f-0942-4dff-af16-d1c0b8dbd981]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[b99e1a06-dd70-46e1-9f24-d5f2ff694f7b] succeeded in 0.0371418920004s: None
Jan 21 04:30:20 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: my server

And still just resetting dropped connection

Checking on /var/lib/pulp/contents/rpm

i can see a bunch of directories and shave rpm packages inside and other no is could be possible is working ?

Took a closer look at the trace and determined my initial diagosis was incorrect. This is related to a know proton bug: https://issues.apache.org/jira/browse/PROTON-1051 that has been fixed. Not goferd. Please disregard the workaround in: https://pulp.plan.io/issues/1557#note-1.