Issue #9362
Updated by fao89 over 2 years ago
**Ticket moved to GitHub**: "pulp/pulp_rpm/2300":https://github.com/pulp/pulp_rpm/issues/2300 ---- Mirroring repositories where the metadata is sha in the yum metadata files fails (when autopublish is enabled). # Environment * Platform: Rockylinux 8 * Pulpcore version: 3.14.5 * Pulp_rpm version: 3.14.2 ALLOWED_CONTENT_CHECKSUMS config in settings.py ~~~ python ALLOWED_CONTENT_CHECKSUMS = ["md5", "sha1", "sha224", "sha256", "sha384", "sha512"] ~~~ Pulp is installed by using the pulpcore Puppet module by TheForeman # Issue description This is currently failing for the yum.puppetlabs.com and rpm.releases.hashicorp.com repositories. The metadata files of those repositories are mentioning *sha* instead of *sha1*, pulp tries to mirror the repositories by setting this to sha, but that's not a valid option in the repository object. The following command was used for synchronizing the repositories with the remotes: ~~~ text pulp rpm repository sync --name mirror-el8-x86_64-hashicorp-stable --remote mirror-el8-x86_64-hashicorp-stable --mirror ~~~ Which gave the following error: ~~~text Task /pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/ failed: 'Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting.' ~~~ ~~~json { "pulp_href": "/pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/", "pulp_created": "2021-09-08T11:58:28.885985Z", "state": "failed", "name": "pulp_rpm.app.tasks.synchronizing.synchronize", "logging_cid": "3ac240218bfc41fcb4d5fa519f8a1732", "started_at": "2021-09-08T11:58:28.954834Z", "finished_at": "2021-09-08T11:58:30.022221Z", "error": { "traceback": " File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n result = func(*args, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 475, in synchronize\n version = dv.create()\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 697, in run\n expected_digests={record_checksum_type: record.checksum},\n File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py\", line 107, in get_downloader\n return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line 410, in get_downloader\n return self.download_factory.build(url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 176, in build\n return builder(download_class, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 208, in _http_or_https\n return download_class(url, **options, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py\", line 59, in __init__\n super().__init__(*args, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 182, in __init__\n super().__init__(url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/pulpcore/download/base.py\", line 113, in __init__\n ).format(self.url)\n", "description": "Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting." }, "worker": null, "parent_task": null, "child_tasks": [], "task_group": null, "progress_reports": [ { "message": "Downloading Metadata Files", "code": "sync.downloading.metadata", "state": "failed", "total": null, "done": 1, "suffix": null }, { "message": "Downloading Artifacts", "code": "sync.downloading.artifacts", "state": "canceled", "total": null, "done": 0, "suffix": null }, { "message": "Associating Content", "code": "associating.content", "state": "canceled", "total": null, "done": 0, "suffix": null } ], "created_resources": [], "reserved_resources_record": [ "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/", "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/" ] } ~~~ The error is the same for the puppet repo shown below. It would be nice if pulp can somehow make the translation from sha to sha1. The remote, repository and distribution objects can be found below (the "hack" by setting the checksum type in the repositories is applied). Did not test it when autopublish was disabled. # How can it be solved It can now be solved by setting the *metadata_checksum_type* and *package_checksum_type* to sha1 in the repositories (and not leaving it blank). # Created remote objects ## Puppetlabs remote ~~~ json { "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/", "pulp_created": "2021-09-08T12:38:31.366488Z", "name": "mirror-el8-x86_64-puppet-7", "url": "https://yum.puppetlabs.com/puppet7/el/8/x86_64/", "ca_cert": null, "client_cert": null, "tls_validation": false, "proxy_url": null, "pulp_labels": {}, "pulp_last_updated": "2021-09-08T14:31:49.112734Z", "download_concurrency": null, "max_retries": null, "policy": "on_demand", "total_timeout": null, "connect_timeout": null, "sock_connect_timeout": null, "sock_read_timeout": null, "headers": [], "rate_limit": null, "sles_auth_token": null }, ~~~ ## Hashicorp remote ~~~ json { "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/", "pulp_created": "2021-09-08T09:26:43.358984Z", "name": "mirror-el8-x86_64-hashicorp-stable", "url": "https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/", "ca_cert": null, "client_cert": null, "tls_validation": false, "proxy_url": null, "pulp_labels": {}, "pulp_last_updated": "2021-09-08T14:31:48.838823Z", "download_concurrency": null, "max_retries": null, "policy": "on_demand", "total_timeout": null, "connect_timeout": null, "sock_connect_timeout": null, "sock_read_timeout": null, "headers": [], "rate_limit": null, "sles_auth_token": null }, ~~~ # Created repository objects ## Puppetlabs repository ~~~ json { "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/", "pulp_created": "2021-09-08T12:38:31.425957Z", "versions_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/", "pulp_labels": {}, "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/1/", "name": "mirror-el8-x86_64-puppet-7", "description": null, "retained_versions": null, "remote": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/", "autopublish": true, "metadata_signing_service": null, "retain_package_versions": 0, "metadata_checksum_type": "sha1", "package_checksum_type": "sha1", "gpgcheck": 0, "repo_gpgcheck": 0, "sqlite_metadata": false }, ~~~ ## Hashicorp repository ~~~ json { "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/", "pulp_created": "2021-09-08T09:26:43.420128Z", "versions_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/", "pulp_labels": {}, "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/1/", "name": "mirror-el8-x86_64-hashicorp-stable", "description": null, "retained_versions": null, "remote": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/", "autopublish": true, "metadata_signing_service": null, "retain_package_versions": 0, "metadata_checksum_type": "sha1", "package_checksum_type": "sha1", "gpgcheck": 0, "repo_gpgcheck": 0, "sqlite_metadata": false }, ~~~ # Distribution objects ## Puppet distribution ~~~ json { "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/d8a10b0f-5a02-4605-820a-da357ae9ef6c/", "pulp_created": "2021-09-08T12:39:17.386142Z", "base_path": "rpm/pub/mirrors/el8/e86_64/puppet/7", "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/puppet/7/", "content_guard": null, "pulp_labels": {}, "name": "mirror-el8-x86_64-puppet-7", "repository": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/", "publication": null }, ~~~ ## Hashicorp distribution ~~~ json { "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/94a101ff-e4f5-4714-a752-732190b166fd/", "pulp_created": "2021-09-08T09:38:24.947840Z", "base_path": "rpm/pub/mirrors/el8/e86_64/hashicorp/stable", "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/hashicorp/stable/", "content_guard": null, "pulp_labels": {}, "name": "mirror-el8-x86_64-hashicorp-stable", "repository": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/", "publication": null }, ~~~
.