Issue #1557: Pulp is no sync anymore after upgrade to foreman 1.10 - Pulp

Actions

Send by e-mail Copy link

Issue #1557

closed

Pulp is no sync anymore after upgrade to foreman 1.10

Added by mgamboa over 8 years ago. Updated over 4 years ago.

Status:

CLOSED - NOTABUG

Priority:

Normal

Assignee:

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

2.6.4

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

I try with foreman bug system and they point me out that is a pulp issue

Short story i just upgrade from 1.9 to 1.10 foreman my smartproxy and now the smartproxy is no sync packages is give this error and is just drop the connection

Jan 19 22:53:06 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1):
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - 'ConnectionClosed' object has no attribute 'url'
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - Traceback (most recent call last):
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in fn
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(*args, **keywords)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 842, in send
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self._impl.send(address, signed, ttl)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 44, in fn
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(messenger, *args, **kwargs)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 64, in _fn
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(sender, *args, **kwargs)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/producer.py", line 99, in send
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - sender = self.connection.sender(address)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 125, in sender
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return self._impl.create_sender(address, name=name)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 203, in create_sender
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return BlockingSender(self, self.container.create_sender(self.conn, utf8(address), name=utf8(name), handler=handler, options=options))
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 67, in init
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - super(BlockingSender, self)._init_(connection, sender)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 36, in init
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - msg="Opening link %s" % link.name)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 231, in wait
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self.container.process()
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3729, in dispatch
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - ev.dispatch(self.handler)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3654, in dispatch
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - result = dispatch(handler, type.method, self)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3543, in dispatch
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return m(*args)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 254, in on_connection_remote_close
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - raise ConnectionClosed(event.connection)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 178, in init
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - txt = "Connection %s closed" % self.url
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - AttributeError: 'ConnectionClosed' object has no attribute 'url'
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - send (progress), failed
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - Traceback (most recent call last):
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/agent/rmi.py", line 239, in report
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - details=self.details)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 44, in _fn
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - raise ModelError(*e.args)
Jan 19 22:53:06 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - ModelError: 'ConnectionClosed' object has no attribute 'url'
Jan 19 22:53:06 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection:
Jan 19 22:53:06 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection:

python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.24-8.pulp.el7.noarch
pulp-katello-0.4-3.el7.noarch
pulp-rpm-handlers-2.6.4-1.el7.noarch
python-pulp-docker-common-1.0.2-1.el7.noarch
python-pulp-puppet-common-2.6.4-1.el7.noarch
python-pulp-bindings-2.6.4-1.el7.noarch
pulp-nodes-child-2.6.4-1.el7.noarch
python-pulp-rpm-common-2.6.4-1.el7.noarch
pulp-server-2.6.4-1.el7.noarch
python-pulp-agent-lib-2.6.4-1.el7.noarch
rubygem-smart_proxy_pulp-1.0.1-2.el7.noarch
pulp-rpm-plugins-2.6.4-1.el7.noarch
pulp-docker-plugins-1.0.2-1.el7.noarch
pulp-puppet-plugins-2.6.4-1.el7.noarch
pulp-nodes-common-2.6.4-1.el7.noarch
python-pulp-common-2.6.4-1.el7.noarch
pulp-selinux-2.6.4-1.el7.noarch

Actions

Copy link

Updated by jortel@redhat.com over 8 years ago

This looks like a known issue that has been fixed in goferd 2.7.1 which is already available in Fedora 23 and Rawhide. I can't find the BZ at the moment.

It goes like this:

1. Send request(s) to an agent.
2. Stop goferd (before it has completed them).
3. Unregister the consumer/capsule.
4. Start goferd.

On restart, goferd will restart the unfinished request(s) which are routed to the katello-agent plugin. However, since the consumer/capsule is no longer registered, it's URL for the broker is 'None'.

A workaround is to do:

1. Purge outstanding requests for the katelloagent (plugin).

  $ cd /var/lib/gofer/messaging/pending/katelloagent
  $ rm *

2. Restart goferd.

A few choices going forward:

1. Back port fix to goferd 2.6.x
2. Switch to goferd 2.7

Actions

Copy link

Updated by jsherril@redhat.com over 8 years ago

Katello is fine upgrading to goferd 2.7.1, are builds available for el5,6,7 for that package?

Actions

Copy link

Updated by jortel@redhat.com over 8 years ago

Not yet available for EL5/6/7 but I can get the ball rolling.

Actions

Copy link

Updated by mgamboa over 8 years ago

I follow the instructions you provide
$ cd /var/lib/gofer/messaging/pending/katelloagent
$ rm *
$ restart goferd

And still having this issues

Jan 21 04:29:18 awscapsule01 goferd: [INFO][pulp.agent.c5ca0064-6e6f-4557-8535-5df729b07580] root:501 - connected to my server:5647
Jan 21 04:29:19 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: my server
Jan 21 04:29:19 awscapsule01 goferd: [INFO][worker-0] gofer.rmi.dispatcher:600 - call: Content.update() sn=bbfa9d76-7160-471c-b1cf-64b741cc1db6 data={'task_id': '774781db-982b-4a47-8a09-fb3ca0584ce1', 'consumer_id': 'c5ca0064-6e6f-4557-8535-5df729b07580'}
Jan 21 04:29:20 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:21 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:21 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:21 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:23 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:24 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:26 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:27 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:28 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:28 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:29 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:30 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:30 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:31 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:31 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:29:31 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: nz2zmsosat01.pp.vficonnect.net.nz
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(*args, **keywords)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 842, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self._impl.send(address, signed, ttl)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 44, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(messenger, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 64, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(sender, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/producer.py", line 99, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - sender = self.connection.sender(address)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 125, in sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return self._impl.create_sender(address, name=name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 203, in create_sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return BlockingSender(self, self.container.create_sender(self.conn, utf8(address), name=utf8(name), handler=handler, options=options))
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 67, ininit_
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - super(BlockingSender, self).__init__(connection, sender)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 36, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - msg="Opening link %s" % link.name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 231, in wait
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self.container.process()
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3729, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - ev.dispatch(self.handler)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3654, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - result = dispatch(handler, type.method, self)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3543, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return m(*args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 254, in on_connection_remote_close
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - raise ConnectionClosed(event.connection)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 178, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - txt = "Connection %s closed" % self.url
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - AttributeError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - send (progress), failed
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/agent/rmi.py", line 239, in report
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - details=self.details)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 44, in fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - raise ModelError(*e.args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - ModelError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(*args, **keywords)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 842, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self._impl.send(address, signed, ttl)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 44, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(messenger, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 64, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return fn(sender, *args, **kwargs)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/producer.py", line 99, in send
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - sender = self.connection.sender(address)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 125, in sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return self._impl.create_sender(address, name=name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 203, in create_sender
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return BlockingSender(self, self.container.create_sender(self.conn, utf8(address), name=utf8(name), handler=handler, options=options))
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 67, ininit_
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - super(BlockingSender, self).__init__(connection, sender)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 36, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - msg="Opening link %s" % link.name)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 231, in wait
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - self.container.process()
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3729, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - ev.dispatch(self.handler)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3654, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - result = dispatch(handler, type.method, self)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/__init__.py", line 3543, in dispatch
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - return m(*args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 254, in on_connection_remote_close
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - raise ConnectionClosed(event.connection)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 178, in init
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - txt = "Connection %s closed" % self.url
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - AttributeError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - send (progress), failed
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - Traceback (most recent call last):
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/agent/rmi.py", line 239, in report
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - details=self.details)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 44, in _fn
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - raise ModelError(*e.args)
Jan 21 04:30:18 awscapsule01 goferd: [ERROR][worker-0] gofer.agent.rmi:241 - ModelError: 'ConnectionClosed' object has no attribute 'url'
Jan 21 04:30:18 awscapsule01 goferd: [INFO][worker-0] pulp_node.handlers.model:475 - Importer: Centos-6/nodes_http_importer, updated
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[1b48a5b1-2d9c-41a1-8b60-16f5b3552148]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.importer.update_importer_config[305e3a0b-8f6f-4c19-87ce-1688735adf5a]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[6d55c27f-c57f-4f8f-a912-1fbff269fa7c]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[1b48a5b1-2d9c-41a1-8b60-16f5b3552148] succeeded in 0.0352381469997s: None
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[b99e1a06-dd70-46e1-9f24-d5f2ff694f7b]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[6913aa59-c51e-4924-bf1d-b6974aef83ce]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[a7d17f8f-0942-4dff-af16-d1c0b8dbd981]
Jan 21 04:30:18 awscapsule01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[b99e1a06-dd70-46e1-9f24-d5f2ff694f7b] succeeded in 0.0371418920004s: None
Jan 21 04:30:20 awscapsule01 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: my server

And still just resetting dropped connection

Checking on /var/lib/pulp/contents/rpm

i can see a bunch of directories and shave rpm packages inside and other no is could be possible is working ?

Actions

Copy link

Updated by jortel@redhat.com over 8 years ago

Took a closer look at the trace and determined my initial diagosis was incorrect. This is related to a know proton bug: https://issues.apache.org/jira/browse/PROTON-1051 that has been fixed. Not goferd. Please disregard the workaround in: https://pulp.plan.io/issues/1557#note-1.

Actions

Copy link