Ticket moved to GitHub: "pulp/pulp_rpm/2300":https://github.com/pulp/pulp_rpm/issues/2300

Mirroring repositories where the metadata is sha in the yum metadata files fails (when autopublish is enabled).

Environment¶

• Platform: Rockylinux 8
• Pulpcore version: 3.14.5
• Pulp_rpm version: 3.14.2

ALLOWED_CONTENT_CHECKSUMS config in settings.py

ALLOWED_CONTENT_CHECKSUMS = ["md5", "sha1", "sha224", "sha256", "sha384", "sha512"]

Pulp is installed by using the pulpcore Puppet module by TheForeman

Issue description¶

This is currently failing for the yum.puppetlabs.com and rpm.releases.hashicorp.com repositories.

The metadata files of those repositories are mentioning sha instead of sha1, pulp tries to mirror the repositories by setting this to sha, but that's not a valid option in the repository object.

The following command was used for synchronizing the repositories with the remotes:

pulp rpm repository sync --name mirror-el8-x86_64-hashicorp-stable --remote mirror-el8-x86_64-hashicorp-stable --mirror

Which gave the following error:

Task /pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/ failed: 'Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting.'

{
  "pulp_href": "/pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/",
  "pulp_created": "2021-09-08T11:58:28.885985Z",
  "state": "failed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "3ac240218bfc41fcb4d5fa519f8a1732",
  "started_at": "2021-09-08T11:58:28.954834Z",
  "finished_at": "2021-09-08T11:58:30.022221Z",
  "error": {
    "traceback": "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 475, in synchronize\n    version = dv.create()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 697, in run\n    expected_digests={record_checksum_type: record.checksum},\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py\", line 107, in get_downloader\n    return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line 410, in get_downloader\n    return self.download_factory.build(url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 176, in build\n    return builder(download_class, url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 208, in _http_or_https\n    return download_class(url, **options, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py\", line 59, in __init__\n    super().__init__(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 182, in __init__\n    super().__init__(url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/base.py\", line 113, in __init__\n    ).format(self.url)\n",
    "description": "Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting."
  },
  "worker": null,
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "failed",
      "total": null,
      "done": 1,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    }
  ],
  "created_resources": [],
  "reserved_resources_record": [
    "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/"
  ]
}

The error is the same for the puppet repo shown below.

It would be nice if pulp can somehow make the translation from sha to sha1.

The remote, repository and distribution objects can be found below (the "hack" by setting the checksum type in the repositories is applied).

Did not test it when autopublish was disabled.

How can it be solved¶

It can now be solved by setting the metadata_checksum_type and package_checksum_type to sha1 in the repositories (and not leaving it blank).

Created remote objects¶

Puppetlabs remote¶

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/",
    "pulp_created": "2021-09-08T12:38:31.366488Z",
    "name": "mirror-el8-x86_64-puppet-7",
    "url": "https://yum.puppetlabs.com/puppet7/el/8/x86_64/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": false,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-09-08T14:31:49.112734Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": [],
    "rate_limit": null,
    "sles_auth_token": null
  },

Hashicorp remote¶

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "pulp_created": "2021-09-08T09:26:43.358984Z",
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "url": "https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": false,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-09-08T14:31:48.838823Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": [],
    "rate_limit": null,
    "sles_auth_token": null
  },

Created repository objects¶

Puppetlabs repository¶

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/",
    "pulp_created": "2021-09-08T12:38:31.425957Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/1/",
    "name": "mirror-el8-x86_64-puppet-7",
    "description": null,
    "retained_versions": null,
    "remote": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/",
    "autopublish": true,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": "sha1",
    "package_checksum_type": "sha1",
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

Hashicorp repository¶

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/",
    "pulp_created": "2021-09-08T09:26:43.420128Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/1/",
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "description": null,
    "retained_versions": null,
    "remote": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "autopublish": true,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": "sha1",
    "package_checksum_type": "sha1",
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

Distribution objects¶

Puppet distribution¶

  {
    "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/d8a10b0f-5a02-4605-820a-da357ae9ef6c/",
    "pulp_created": "2021-09-08T12:39:17.386142Z",
    "base_path": "rpm/pub/mirrors/el8/e86_64/puppet/7",
    "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/puppet/7/",
    "content_guard": null,
    "pulp_labels": {},
    "name": "mirror-el8-x86_64-puppet-7",
    "repository": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/",
    "publication": null
  },

Hashicorp distribution¶

  {
    "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/94a101ff-e4f5-4714-a752-732190b166fd/",
    "pulp_created": "2021-09-08T09:38:24.947840Z",
    "base_path": "rpm/pub/mirrors/el8/e86_64/hashicorp/stable",
    "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/hashicorp/stable/",
    "content_guard": null,
    "pulp_labels": {},
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "repository": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/",
    "publication": null
  },