Project

Profile

Help

Issue #6762

closed

Cannot sync a remote that's using a x509 content guard

Added by david.macneil@actual-experience.com almost 4 years ago. Updated almost 4 years ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

We have an RPM respository that 's protected with an x509 content guard certificate. If I try to set up syncing with this repository from another pulp instance, to ensure we have an active backup / failover I am not able to get the 2nd instance to sync as it isn't setting the expected header SSL-CLIENT-CERTIFICATE to authenticate with the content guard plugin.

Here's the repository I'm trying to sync

root@pulp-repo-1[shared_services]:centos$ http $AD/pulp/api/v3/distributions/rpm/rpm/
HTTP/1.1 200 OK
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 520
Content-Type: application/json
Date: Mon, 18 May 2020 15:56:09 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1, 
    "next": null, 
    "previous": null, 
    "results": [
        {
            "base_path": "actual-amazon-linux", 
            "base_url": "http://pulp.actual-experience.com:24816/pulp/content/actual-amazon-linux", 
            "content_guard": "/pulp/api/v3/contentguards/certguard/x509/55256924-2315-435c-b6ed-c935e5c1365a/", 
            "name": "actual-amazon-linux", 
            "publication": "/pulp/api/v3/publications/rpm/rpm/af560afc-2c2e-47ef-8465-875e1341a083/", 
            "pulp_created": "2020-02-26T13:19:10.195866Z", 
            "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/55eb7ce4-1087-4ab0-a5ba-26d63a013fd2/"
        }
    ]
}
[root@ip-172-32-0-97 pulp]# http POST $AD/pulp/api/v3/remotes/rpm/rpm/ name=pulp-replicate url=http://pulp.actual-experience.com/pulp/content/actual-amazon-linux/ policy=immediate client_cert=@~/cert.pem
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 477
Content-Type: application/json
Date: Mon, 18 May 2020 16:05:36 GMT
Location: /pulp/api/v3/remotes/rpm/rpm/c5cf0a5c-4c84-4fd4-88ff-82d4862813ea/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "ca_cert": null, 
    "client_cert": "36e8286f4f4f48b502fe7722b19057db2aa7807f6ee267c79406a8048ccb6d05", 
    "client_key": null, 
    "download_concurrency": 20, 
    "name": "pulp-replicate", 
    "policy": "immediate", 
    "proxy_url": null, 
    "pulp_created": "2020-05-18T16:05:36.823449Z", 
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/c5cf0a5c-4c84-4fd4-88ff-82d4862813ea/", 
    "pulp_last_updated": "2020-05-18T16:05:36.823470Z", 
    "tls_validation": true, 
    "url": "http://pulp.actual-experience.com/pulp/content/actual-amazon-linux/"
}

[root@ip-172-32-0-97 pulp]# http POST $AD/pulp/api/v3/repositories/rpm/rpm/acc60ae5-7fac-42bf-bf9d-d843a35952ad/sync/ remote=/pulp/api/v3/remotes/rpm/rpm/c5cf0a5c-4c84-4fd4-88ff-82d4862813ea/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 18 May 2020 16:06:05 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/50330d7d-d932-4618-815b-38d8e0e8db1b/"
}

[root@ip-172-32-0-97 pulp]# http $AD/pulp/api/v3/tasks/50330d7d-d932-4618-815b-38d8e0e8db1b/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2272
Content-Type: application/json
Date: Mon, 18 May 2020 16:06:14 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [], 
    "created_resources": [], 
    "error": {
        "description": "403, message='\\'HTTP header \"SSL-CLIENT-CERTIFICATE\" not found.\\'', url=URL('https://pulp.actual-experience.com:443/pulp/content/actual-amazon-linux/.treeinfo')", 
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 129, in synchronize\n    treeinfo = get_treeinfo_data(remote)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/kickstart/treeinfo.py\", line 24, in get_treeinfo_data\n    result = downloader.fetch()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 154, in fetch\n    return done.pop().result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 185, in _run\n    response.raise_for_status()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 946, in raise_for_status\n    headers=self.headers)\n"
    }, 
    "finished_at": "2020-05-18T16:06:06.142745Z", 
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize", 
    "parent_task": null, 
    "progress_reports": [], 
    "pulp_created": "2020-05-18T16:06:05.922233Z", 
    "pulp_href": "/pulp/api/v3/tasks/50330d7d-d932-4618-815b-38d8e0e8db1b/", 
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/c5cf0a5c-4c84-4fd4-88ff-82d4862813ea/", 
        "/pulp/api/v3/repositories/rpm/rpm/acc60ae5-7fac-42bf-bf9d-d843a35952ad/"
    ], 
    "started_at": "2020-05-18T16:06:06.026583Z", 
    "state": "failed", 
    "task_group": null, 
    "worker": "/pulp/api/v3/workers/2e876379-b7be-44aa-92ba-d21fb08cda16/"
}

I can verify that the certificate being used above is valid given the same request as is failing in the above output:

[root@ip-172-32-0-97 pulp]# curl -k https://pulp.actual-experience.com/pulp/content/actual-amazon-linux/.treeinfo -H "SSL-CLIENT-CERTIFICATE:$(cat ~/cert.pem | tr -d '\n')"
404: Not Found[root@ip-172-32-0-97 pulp]#

As a result, it doesn't seem possible to mirror or otherwise implement any sort of HA configuration when using the content guard plugin to protect that repository at the same time and given that this seems to be the advised method for replication and redundancy in the docs:

https://pulpproject.org/about-pulp-3/

  • Pulp 2’s nodes concept has been removed in favor of Pulp server-to-server syncing

It would be a real improvement if the remote API had a field to set the appropriate header expected by the content guard plugin:

https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create

Actions #1

Updated by fao89 almost 4 years ago

  • Project changed from Pulp to CertGuard
  • Triaged changed from No to Yes
Actions #2

Updated by bmbouter almost 4 years ago

The original implementation of certguard used the SSL-CLIENT-CERTIFICATE but this was problematic for exactly the reasons you're running into. The implementation was switched to having the cert arrive via the TLS connection itself with this ticket https://pulp.plan.io/issues/6352 Those changes are released today 0.1.0rc5.

The downside is that the migrations had to be remade from scratch due to technical reasons. So for you to upgrade to the latest, you'll have to remove your existing use of certguard, delete certguard tables from your DB, and reinstall certguard 0.1.0rc5 (just released) and reapply your migrations. Reach out on pulp-list, comment here, or come to #pulp on irc for some help doing ^ if you like.

As of now 0.1.0rc5 is the latest version and likely will become the GA here in 2-3 weeks, so I only expect you to go through this huge hassle once. After that you should be able to use Pulp to sync via the Remote.client_cert and Remote.client_key features of Pulp. I filed a ticket for us to add a functional test for this, but I believe it works already. https://pulp.plan.io/issues/6810 That outlines roughly the procedure you should use.

Let me know how it goes, and I regret you not having a clean upgrade path for this. Any feedback is welcome.

Actions #3

Updated by david.macneil@actual-experience.com almost 4 years ago

Many thanks for the update on this. We have, for now, worked around the issue but I'll let you know if we have reason / chance to test what you've suggested. We'll probably wait for the next time we want to perform an upgrade however so hopefully there will be a stable upgrade path at that time. The reason for this migration was to upgrade from pulpcore 3.1.1 to 3.3.1 make use of the s3 plugin.

Actions #4

Updated by bmbouter almost 4 years ago

  • Status changed from NEW to CLOSED - WORKSFORME

I believe this working so I'm going to close as WORKSFORME. Please comment if there is something we should do to make this better. Thank you for filing it.

Actions #5

Updated by bmbouter almost 4 years ago

wrote:

Many thanks for the update on this. We have, for now, worked around the issue but I'll let you know if we have reason / chance to test what you've suggested. We'll probably wait for the next time we want to perform an upgrade however so hopefully there will be a stable upgrade path at that time. The reason for this migration was to upgrade from pulpcore 3.1.1 to 3.3.1 make use of the s3 plugin.

Sounds good. If you can start from a GA release there will absolutely be an upgrade path. Unfortunately if you're using <= 0.1.0rc3 you'll have to uninstall + drop certguard tables + install.

Way to go with the S3 usage! If there is anything we can help with let us know.

Also available in: Atom PDF