Project

Profile

Help

Issue #9362

Mirroring rpm repositories with sha1 metatdata fails

Added by maartenb 7 days ago. Updated 1 day ago.

Status:
ASSIGNED
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Q4-2021

Description

Mirroring repositories where the metadata is sha in the yum metadata files fails (when autopublish is enabled).

Environment

  • Platform: Rockylinux 8
  • Pulpcore version: 3.14.5
  • Pulp_rpm version: 3.14.2

ALLOWED_CONTENT_CHECKSUMS config in settings.py

ALLOWED_CONTENT_CHECKSUMS = ["md5", "sha1", "sha224", "sha256", "sha384", "sha512"]

Pulp is installed by using the pulpcore Puppet module by TheForeman

Issue description

This is currently failing for the yum.puppetlabs.com and rpm.releases.hashicorp.com repositories.

The metadata files of those repositories are mentioning sha instead of sha1, pulp tries to mirror the repositories by setting this to sha, but that's not a valid option in the repository object.

The following command was used for synchronizing the repositories with the remotes:

pulp rpm repository sync --name mirror-el8-x86_64-hashicorp-stable --remote mirror-el8-x86_64-hashicorp-stable --mirror

Which gave the following error:

Task /pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/ failed: 'Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting.'
{
  "pulp_href": "/pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/",
  "pulp_created": "2021-09-08T11:58:28.885985Z",
  "state": "failed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "3ac240218bfc41fcb4d5fa519f8a1732",
  "started_at": "2021-09-08T11:58:28.954834Z",
  "finished_at": "2021-09-08T11:58:30.022221Z",
  "error": {
    "traceback": "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 475, in synchronize\n    version = dv.create()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 697, in run\n    expected_digests={record_checksum_type: record.checksum},\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py\", line 107, in get_downloader\n    return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line 410, in get_downloader\n    return self.download_factory.build(url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 176, in build\n    return builder(download_class, url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 208, in _http_or_https\n    return download_class(url, **options, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py\", line 59, in __init__\n    super().__init__(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 182, in __init__\n    super().__init__(url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/base.py\", line 113, in __init__\n    ).format(self.url)\n",
    "description": "Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting."
  },
  "worker": null,
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "failed",
      "total": null,
      "done": 1,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    }
  ],
  "created_resources": [],
  "reserved_resources_record": [
    "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/"
  ]
}

The error is the same for the puppet repo shown below.

It would be nice if pulp can somehow make the translation from sha to sha1.

The remote, repository and distribution objects can be found below (the "hack" by setting the checksum type in the repositories is applied).

Did not test it when autopublish was disabled.

How can it be solved

It can now be solved by setting the metadata_checksum_type and package_checksum_type to sha1 in the repositories (and not leaving it blank).

Created remote objects

Puppetlabs remote

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/",
    "pulp_created": "2021-09-08T12:38:31.366488Z",
    "name": "mirror-el8-x86_64-puppet-7",
    "url": "https://yum.puppetlabs.com/puppet7/el/8/x86_64/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": false,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-09-08T14:31:49.112734Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": [],
    "rate_limit": null,
    "sles_auth_token": null
  },

Hashicorp remote

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "pulp_created": "2021-09-08T09:26:43.358984Z",
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "url": "https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": false,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-09-08T14:31:48.838823Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": [],
    "rate_limit": null,
    "sles_auth_token": null
  },

Created repository objects

Puppetlabs repository

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/",
    "pulp_created": "2021-09-08T12:38:31.425957Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/1/",
    "name": "mirror-el8-x86_64-puppet-7",
    "description": null,
    "retained_versions": null,
    "remote": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/",
    "autopublish": true,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": "sha1",
    "package_checksum_type": "sha1",
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

Hashicorp repository

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/",
    "pulp_created": "2021-09-08T09:26:43.420128Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/1/",
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "description": null,
    "retained_versions": null,
    "remote": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "autopublish": true,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": "sha1",
    "package_checksum_type": "sha1",
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

Distribution objects

Puppet distribution

  {
    "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/d8a10b0f-5a02-4605-820a-da357ae9ef6c/",
    "pulp_created": "2021-09-08T12:39:17.386142Z",
    "base_path": "rpm/pub/mirrors/el8/e86_64/puppet/7",
    "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/puppet/7/",
    "content_guard": null,
    "pulp_labels": {},
    "name": "mirror-el8-x86_64-puppet-7",
    "repository": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/",
    "publication": null
  },

Hashicorp distribution

  {
    "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/94a101ff-e4f5-4714-a752-732190b166fd/",
    "pulp_created": "2021-09-08T09:38:24.947840Z",
    "base_path": "rpm/pub/mirrors/el8/e86_64/hashicorp/stable",
    "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/hashicorp/stable/",
    "content_guard": null,
    "pulp_labels": {},
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "repository": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/",
    "publication": null
  },

History

#1 Updated by dalley 7 days ago

  • Triaged changed from No to Yes
  • Quarter set to Q4-2021

#2 Updated by dalley 1 day ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

Please register to edit this issue

Also available in: Atom PDF