Pulp

Does this error still occur after restarting httpd?

+ This comment was cloned from Bugzilla #1142273 comment 1 +

Yes, restarting httpd on server and/or child node have no effect.
Restarting goferd on child node have no effect either.

Firewalls turned of and SELinux on permissive - no change.

I have now made a completely new test with new installs of parent and child with the same result - node sync does not work.

+ This comment was cloned from Bugzilla #1142273 comment 2 +

Here is some results of node sync

After doing all the config changes and restarting services, I did get the same error.
[root@qe-blade-05 ~]# pulp-admin node sync run --node-id node1
This command may be exited via ctrl+c without affecting the request.

Error occurred during synchronization, check the child node logs for details

--------------------------------------------------------------------
Child Node Synchronization
--------------------------------------------------------------------

Repository:
Action: Pending
Content Sources:
Downloads:
Id: zoo
Units:
Added: 0
Removed: 0
Updated: 0

--------------------------------------------------------------------
The following [1] errors were reported
--------------------------------------------------------------------

- 01: An unexpected error occurred. repo_id=None

[root@qe-blade-05 ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

Restarting httpd on server did not seem to fix it.

[root@qe-blade-05 ~]# pulp-admin node sync run --node-id node1
This command may be exited via ctrl+c without affecting the request.

Error occurred during synchronization, check the child node logs for details

--------------------------------------------------------------------
Child Node Synchronization
--------------------------------------------------------------------

Repository:
Action: Pending
Content Sources:
Downloads:
Id: zoo
Units:
Added: 0
Removed: 0
Updated: 0

--------------------------------------------------------------------
The following [1] errors were reported
--------------------------------------------------------------------

- 01: An unexpected error occurred. repo_id=None

Now I ran the following on the node child

[root@mgmt12 ~]# service goferd restart
Stopping goferd[ OK ]
Starting goferd[ OK ]

That did not fix the issue

Then I restarted httpd on the child and that fixed the issue and got the sync going
[root@mgmt12 ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

[root@qe-blade-05 ~]# pulp-admin node sync run --node-id node1
This command may be exited via ctrl+c without affecting the request.

(1/1) Repository: zoo
[==================================================] 100%

Synchronization succeeded

--------------------------------------------------------------------
Child Node Synchronization
--------------------------------------------------------------------

Repository:
Action: Added
Content Sources:
Downloads:
Source Id: Parent Node
Total Failed: 0
Total Succeeded: 32
Total Passes: 1
Total Sources: 0
Id: zoo
Units:
Added: 39
Removed: 0
Updated: 0

[root@qe-blade-05 ~]#

+ This comment was cloned from Bugzilla #1142273 comment 3 +

Unfortunately that solution did not work for me, i still get the same failed result. tried multiple times to restart services in different order, reboots... no change.

I have now tested this on two new VM'w with CentOS 7 and get the same results.

On the CentOS VM's pulp-server-2.4.2-1.el7

Logs from the new CentOS VM'S:

Server logs:
Sep 24 21:09:14 pulp pulp: pulp.server.agent.direct.services:INFO: {'reboot': {'scheduled': False, 'details': {}}, 'details': {'node': {'details': {'errors': [{'error_id': 'exception', 'details': {'message': 'RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed', 'repo_id': None}}], 'repositories': [{'action': 'pending', 'sources': {}, 'repo_id': 'rhel-7-pulp', 'units': {'removed': 0, 'added': 0, 'updated': 0}}]}, 'succeeded': False}}, 'succeeded': False, 'num_changes': 0}

Child Node logs:

Sep 24 21:09:14 pulp2 pulp: pulp.server.webservices.middleware.exception:ERROR: raise AuthenticationFailed(auth_utils.CODE_INVALID_SSL_CERT)
Sep 24 21:09:14 pulp2 pulp: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: Pulp exception occurred: AuthenticationFailed
Sep 24 21:09:14 pulp2 goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - synchronization failed
Sep 24 21:09:14 pulp2 goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line 116, in synchronize validator.validate(request.bindings) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 70, in _validate_plugins child = ChildServer() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 88, in init self.importers = self._importers() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 99, in _importers http = bindings.server_info.get_importers() File "/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 43, in get_importers return self.server.GET (path) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 93, in GET return self._request('GET', path, queries) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 192, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed
Sep 24 21:09:14 pulp2 goferd: [ERROR][worker-0] pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None

+ This comment was cloned from Bugzilla #1142273 comment 4 +

I'm having the same issue, fresh child node install:

Verified oauth is working

Verify ssl is off in nodes.conf
]# cat /etc/pulp/nodes.conf |grep veri
verify_ssl: false

restarting httpd and goferd...

unregistered, registered, activated client, and then bound to single repo:

pulp-admin node list
--------------------------------------------------------------------
Child Nodes
--------------------------------------------------------------------

Id: node-1
Display Name: node-1
Description: None
Bindings:
Additive: pulp
Update Strategy: additive
Notes:

pulp-admin node sync run --node-id node-1
This command may be exited via ctrl+c without affecting the request.

Error occurred during synchronization, check the child node logs for details

--------------------------------------------------------------------
Child Node Synchronization
--------------------------------------------------------------------

Repository:
Action: Pending
Content Sources:
Downloads:
Id: pulp
Units:
Added: 0
Removed: 0
Updated: 0

--------------------------------------------------------------------
The following [1] errors were reported
--------------------------------------------------------------------

- 01: An unexpected error occurred. repo_id=None

Logs:

Parent
=> /var/log/messages <==
2014-09-25T14:50:31.600352+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: Task RMI (succeeded)
2014-09-25T14:50:31.600462+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: Succeeded
2014-09-25T14:50:31.600528+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: sn : 4a6a36e0-4c83-402f-a36f-0b7c16a2c42d
2014-09-25T14:50:31.600680+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: origin : dd65c874-a4b0-42ca-8741-8d703323bae8
2014-09-25T14:50:31.600908+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: user data : {'task_id': 'd9063314-51e6-4b1a-8b60-9c16b74655b0', 'consumer_id': 'vs-prod-plp01.mgmt.prod.uc'}
2014-09-25T14:50:31.600983+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: retval:
2014-09-25T14:50:31.601055+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: {'reboot': {'scheduled': False, 'details': {}}, 'details': {'node': {'details': {'errors': [{'error_id': 'exception', 'details': {'message': 'RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed', 'repo_id': None}}], 'repositories': [{'action': 'pending', 'sources': {}, 'repo_id': 'pulp', 'units': {'removed': 0, 'added': 0, 'updated': 0}}]}, 'succeeded': False}}, 'succeeded': False, 'num_changes': 0}

Child
==> /var/log/messages <==
2014-09-25T14:53:08.609463+01:00 fake_servername goferd: [INFO][worker-0] gofer.rmi.dispatcher:612 - call: Content.update() sn=af9652fa-b4f3-4c2d-a464-d2891b25ad56 info={'task_id': 'fcf032d2-b213-4169-916c-80ac63dad44a', 'consumer_id': 'fake_servername.mgmt.prod.uc'}
2014-09-25T14:53:08.732310+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: Pulp exception occurred: AuthenticationFailed
2014-09-25T14:53:08.732536+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):
2014-09-25T14:53:08.732899+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in call
2014-09-25T14:53:08.733194+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response)
2014-09-25T14:53:08.733484+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in call
2014-09-25T14:53:08.733774+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response)
2014-09-25T14:53:08.734120+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
2014-09-25T14:53:08.734409+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.handle_with_processors()
2014-09-25T14:53:08.734693+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
2014-09-25T14:53:08.735004+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors)
2014-09-25T14:53:08.735243+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
2014-09-25T14:53:08.735543+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.735784+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
2014-09-25T14:53:08.736058+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return handler()
2014-09-25T14:53:08.736297+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
2014-09-25T14:53:08.736532+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.736766+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
2014-09-25T14:53:08.737040+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.737278+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
2014-09-25T14:53:08.737514+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler()
2014-09-25T14:53:08.737747+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
2014-09-25T14:53:08.738022+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.738259+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
2014-09-25T14:53:08.738494+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle()
2014-09-25T14:53:08.738726+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
2014-09-25T14:53:08.739005+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args)
2014-09-25T14:53:08.739242+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate
2014-09-25T14:53:08.739476+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return f()
2014-09-25T14:53:08.739711+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda>
2014-09-25T14:53:08.739990+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: f = lambda: self._delegate_sub_application(pat, what)
2014-09-25T14:53:08.740229+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application
2014-09-25T14:53:08.740464+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return app.handle_with_processors()
2014-09-25T14:53:08.740700+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
2014-09-25T14:53:08.740978+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors)
2014-09-25T14:53:08.741216+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
2014-09-25T14:53:08.741451+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.741685+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
2014-09-25T14:53:08.741960+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return handler()
2014-09-25T14:53:08.742197+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
2014-09-25T14:53:08.742430+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.742664+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
2014-09-25T14:53:08.742941+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.743177+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
2014-09-25T14:53:08.743412+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler()
2014-09-25T14:53:08.743647+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
2014-09-25T14:53:08.743920+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
2014-09-25T14:53:08.744155+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
2014-09-25T14:53:08.744390+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle()
2014-09-25T14:53:08.744624+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
2014-09-25T14:53:08.744895+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args)
2014-09-25T14:53:08.745132+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
2014-09-25T14:53:08.745368+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return handle_class(cls)
2014-09-25T14:53:08.745603+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
2014-09-25T14:53:08.745871+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return tocall(*args)
2014-09-25T14:53:08.746097+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 203, in _auth_decorator
2014-09-25T14:53:08.746335+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: raise AuthenticationFailed(auth_utils.CODE_FAILED)
2014-09-25T14:53:08.746570+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: Pulp exception occurred: AuthenticationFailed
2014-09-25T14:53:08.750747+01:00 fake_servername goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - synchronization failed
2014-09-25T14:53:08.750816+01:00 fake_servername goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/handlers/strategies.py", line 116, in synchronize validator.validate(request.bindings) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 70, in _validate_plugins child = ChildServer() File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 88, in init self.importers = self._importers() File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 99, in _importers http = bindings.server_info.get_importers() File "/usr/lib/python2.6/site-packages/pulp/bindings/server_info.py", line 43, in get_importers return self.server.GET (path) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 93, in GET return self._request('GET', path, queries) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 192, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed
2014-09-25T14:53:08.763471+01:00 fake_servername goferd: [ERROR][worker-0] pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None
2014-09-25T14:53:08.764569+01:00 fake_servername goferd: [INFO][worker-0] gofer.agent.rmi:149 - sn=af9652fa-b4f3-4c2d-a464-d2891b25ad56 processed in: 176 (ms)

+ This comment was cloned from Bugzilla #1142273 comment 5 +

(In reply to Johan from comment #0)

Description of problem:
Fresh default installation of pulp-server acting as a node to another
default pulp-server both RHEL 7.
Followed instructions:
http://pulp-user-guide.readthedocs.org/en/latest/installation.html
http://pulp-user-guide.readthedocs.org/en/latest/nodes.html

Default certs and verify_ssl=False

Consumer Client works on the node server and can use yum.
pulp-admin and pulp-consumer works on both servers.

pulp-admin node sync run --node-id repo fails every time, tried every thing
including fresh installations with same result.

Also tried pulp-consumer update --keys and restarting goferd - same results.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:
1. Install two RHEL 7 Servers with default installation of pulp-server and
create a repo on one.
2. Configure one as a node to the other using current documentation and try
to sync the repo.
3.

Actual results:

Node sync failure

Expected results:

Node sync success

Additional info:

Log from messages:

Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: Pulp exception occurred:
AuthenticationFailed
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent
call last):
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/
exception.py", line 44, in call
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
self.app(environ, start_response)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/
postponed.py", line 42, in call
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
self.app(environ, start_response)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 279, in wsgi
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: result =
self.handle_with_processors()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 29, in _handle_with_processors
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
process(self.processors)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in process
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return handler()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in <lambda>
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in process
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: result = handler()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in <lambda>
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 28, in process
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return self.handle()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
self._delegate(fn, self.fvars, args)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 422, in _delegate
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return f()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 430, in <lambda>
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: f = lambda:
self._delegate_sub_application(pat, what)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 455, in
_delegate_sub_application
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
app.handle_with_processors()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 29, in _handle_with_processors
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
process(self.processors)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in process
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return handler()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in <lambda>
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in process
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: result = handler()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 26, in <lambda>
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return p(lambda:
process(processors))
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 28, in process
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return self.handle()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
self._delegate(fn, self.fvars, args)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 420, in _delegate
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return
handle_class(cls)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/web/application.py", line 396, in
handle_class
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: return tocall(*args)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/
decorators.py", line 192, in _auth_decorator
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: userid, is_consumer
= authenticate_user()
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/
decorators.py", line 111, in oauth_authentication
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: raise
AuthenticationFailed(auth_utils.CODE_INVALID_SSL_CERT)
Sep 15 10:42:05 repo pulp:
pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed:
Pulp exception occurred: AuthenticationFailed
Sep 15 10:42:05 repo goferd: [ERROR][worker-0]
pulp_node.handlers.strategies:129 - synchronization failed
Sep 15 10:42:05 repo goferd: [ERROR][worker-0]
pulp_node.handlers.strategies:129 - Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line
116, in synchronize validator.validate(request.bindings) File
"/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line
54, in validate self.report.errors.extend(self._validate_plugins(bindings))
File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py",
line 70, in _validate_plugins child = ChildServer() File
"/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line
88, in init self.importers = self._importers() File
"/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line
99, in _importers http = bindings.server_info.get_importers() File
"/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 43, in
get_importers return self.server.GET (path) File
"/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 93, in GET
return self._request('GET', path, queries) File
"/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 151, in
_request self._handle_exceptions(response_code, response_body) File
"/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 192, in
_handle_exceptions raise code_class_mappings[response_code](response_body)
PermissionsException: RequestException: GET request on
/pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred:
AuthenticationFailed
Sep 15 10:42:05 repo goferd: [ERROR][worker-0]
pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None
Sep 15 10:42:05 repo goferd: [INFO][worker-0] gofer.agent.rmi:149 -
sn=7b3315f2-ec18-4ced-be8b-2d714ae0b73b processed in: 114 (ms)

Hello,

I have exactly the same problem with a child on RHEL7 and a parent on RHEL6.

I'm using pulp 2.5, the latest version.

I tried a lot of settings, as you.

Have you find a solution or a new direction to search ?

Thank you.

+ This comment was cloned from Bugzilla #1142273 comment 6 +

Has there been any progress on what is causing this? I've hit the same issue with parent and child on CentOS6, pulp 2.5.

+ This comment was cloned from Bugzilla #1142273 comment 7 +

I was able to repro this issue on Pulp 2.5 and 2.6.

The trick here is that if you copy your settings from [oauth] in server.conf to nodes.conf's [parent_oauth], the key changes.

For example, your server.conf may contain

[oauth]
enabled: true
oauth_key: my_key
oauth_secret: my_secret

However, parent_oauth in nodes.conf needs to say

[parent_oauth]
key: my_key
secret: my_secret
user_id: admin

Note the lack of "oauth_" for the values here! If you copy/paste, you'll get the error described in this issue.

I will update the documentation and conf file to be very explicit about this.