Issue #2929
closedNode sync is failing with unexpected error repo_id=None
Description
Hello,
I am facing this issue wherein when doing a node sync, it throws me an error:
[root@pulpserver ~]# pulp-admin node sync run --node-id child-1
Warning: this Nodes command is deprecated as the Nodes functionality will be
removed from Pulp 3.0.
This command may be exited via ctrl+c without affecting the request.
Error occurred during synchronization, check the child node logs for details
--------------------------------------------------------------------
Child Node Synchronization
--------------------------------------------------------------------
Repository:
Action: Pending
Content Sources:
Downloads:
Id: myown
Units:
Added: 0
Removed: 0
Updated: 0
--------------------------------------------------------------------
The following [1] errors were reported
--------------------------------------------------------------------
- 01: An unexpected error occurred. repo_id=None
A verbose output can be found here: http://paste.ubuntu.com/25155828/
Please note that I have set verify = false in nodes.conf, admin.conf as well as consumer.conf. Hence, I do not see how I should be getting this error. Tried restarting httpd , goferd etc. but to no avail.
I can find these errors in journalctl and /var/log/messages:
Jul 23 23:27:37 pulp-slave-server pulp: pulp.server.webservices.middleware.exception:INFO: Authentication with username None failed: invalid SSL certificate.
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - synchronization failed
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - Traceback (most recent call last):
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line 99, in synchronize
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - validator.validate(request.bindings)
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 38, in validate
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - self.report.errors.extend(self._validate_plugins(bindings))
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 54, in _validate_plugins
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - child = ChildServer()
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 72, in init
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - self.importers = self._importers()
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 83, in _importers
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - http = bindings.server_info.get_importers()
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 31, in get_importers
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - return self.server.GET (path)
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 92, in GET
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - return self._request('GET', path, queries, ignore_prefix=ignore_prefix)
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 166, in _request
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - self._handle_exceptions(response_code, response_body)
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 207, in _handle_exceptions
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - raise code_class_mappings[response_code](response_body)
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.strategies:112 - PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Authentication with username None failed: invalid SSL certificate.
Jul 23 23:27:37 pulp-slave-server goferd: [ERROR][worker-0] pulp_node.handlers.handler:96 - An unexpected error occurred. repo_id=None
[root@pulpserver ~]# rpm -qa pulp-server
pulp-server-2.13.3-1.el7.noarch
On doing a little research, I found out that quite a few users have already reported the bug here in previous versions of pulp: https://pulp.plan.io/issues/528
Please help me fix the issue at the earliest.
Regards,
Rohan Dsouza
Updated by dsouza.rohan06 over 6 years ago
Forgot to mention about oauth. oauth keys and secrets are connect on the parent nodes and the child nodes. Also, a typo above, verify_ssl = false is set in nodes.conf
Updated by ttereshc over 6 years ago
- Tags deleted (
Known Issue for Release Notes)
Updated by bizhang over 6 years ago
Rohan,
Can you validate that your nodes.conf is not running into this issue: https://pulp.plan.io/issues/528#note-10
If that's not the problem, could you share your server.conf and nodes.conf?
Updated by dsouza.rohan06 over 6 years ago
Hi,
I'm pretty sure there is no issues with my config. However, please find details as requested.
server.conf (On Parent node)
[oauth]
enabled: true
oauth_key: ab3cd9j4ks73hf7g
oauth_secret: qwedsa
nodes.conf (On Child node)
[main]
- ca_path: /etc/pki/tls/certs/ca-bundle.crt
- node_certificate: /etc/pki/pulp/nodes/node.crt
verify_ssl = False
- The oauth configuration used to authenticate to this node
- user_id - The pulp user ID
[oauth]
user_id: admin
- The oauth configuration used to authenticate to the parent node
- key - The oauth key
- secret - The oauth secret
- user_id - The pulp user ID
[parent_oauth]
key: ab3cd9j4ks73hf7g
secret: qwedsa
user_id: admin
Kindly help me fix this issue.
Regards,
Rohan Dsouza
Updated by dsouza.rohan06 over 6 years ago
Also, just an FYI, selinux and firewalld on both nodes are disabled.
Updated by dsouza.rohan06 over 6 years ago
Guys, please ignore. I forgot to mention the oauth settings in my child nodes server.conf file. Its working now.
Updated by bmbouter over 6 years ago
- Status changed from NEW to CLOSED - NOTABUG
Thanks for the followup. I'm glad it's working for you. I'm closing as NOTABUG.