Project

Profile

Help

Issue #9362

Mirroring rpm repositories with sha1 metatdata fails

Added by maartenb 3 months ago. Updated about 2 months ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Q4-2021

Description

Mirroring repositories where the metadata is sha in the yum metadata files fails (when autopublish is enabled).

Environment

  • Platform: Rockylinux 8
  • Pulpcore version: 3.14.5
  • Pulp_rpm version: 3.14.2

ALLOWED_CONTENT_CHECKSUMS config in settings.py

ALLOWED_CONTENT_CHECKSUMS = ["md5", "sha1", "sha224", "sha256", "sha384", "sha512"]

Pulp is installed by using the pulpcore Puppet module by TheForeman

Issue description

This is currently failing for the yum.puppetlabs.com and rpm.releases.hashicorp.com repositories.

The metadata files of those repositories are mentioning sha instead of sha1, pulp tries to mirror the repositories by setting this to sha, but that's not a valid option in the repository object.

The following command was used for synchronizing the repositories with the remotes:

pulp rpm repository sync --name mirror-el8-x86_64-hashicorp-stable --remote mirror-el8-x86_64-hashicorp-stable --mirror

Which gave the following error:

Task /pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/ failed: 'Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting.'
{
  "pulp_href": "/pulp/api/v3/tasks/7a00f67b-6ffd-4bfb-bf23-8d7baec51466/",
  "pulp_created": "2021-09-08T11:58:28.885985Z",
  "state": "failed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "3ac240218bfc41fcb4d5fa519f8a1732",
  "started_at": "2021-09-08T11:58:28.954834Z",
  "finished_at": "2021-09-08T11:58:30.022221Z",
  "error": {
    "traceback": "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 475, in synchronize\n    version = dv.create()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 697, in run\n    expected_digests={record_checksum_type: record.checksum},\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py\", line 107, in get_downloader\n    return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line 410, in get_downloader\n    return self.download_factory.build(url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 176, in build\n    return builder(download_class, url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/factory.py\", line 208, in _http_or_https\n    return download_class(url, **options, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py\", line 59, in __init__\n    super().__init__(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 182, in __init__\n    super().__init__(url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/download/base.py\", line 113, in __init__\n    ).format(self.url)\n",
    "description": "Content at the url https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/repodata/cd44a81284ab5a0d27880a2db23755ad47e74945-other.xml.gz does not contain at least one trusted hasher which is specified in 'ALLOWED_CONTENT_CHECKSUMS' setting."
  },
  "worker": null,
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "failed",
      "total": null,
      "done": 1,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    }
  ],
  "created_resources": [],
  "reserved_resources_record": [
    "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/"
  ]
}

The error is the same for the puppet repo shown below.

It would be nice if pulp can somehow make the translation from sha to sha1.

The remote, repository and distribution objects can be found below (the "hack" by setting the checksum type in the repositories is applied).

Did not test it when autopublish was disabled.

How can it be solved

It can now be solved by setting the metadata_checksum_type and package_checksum_type to sha1 in the repositories (and not leaving it blank).

Created remote objects

Puppetlabs remote

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/",
    "pulp_created": "2021-09-08T12:38:31.366488Z",
    "name": "mirror-el8-x86_64-puppet-7",
    "url": "https://yum.puppetlabs.com/puppet7/el/8/x86_64/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": false,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-09-08T14:31:49.112734Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": [],
    "rate_limit": null,
    "sles_auth_token": null
  },

Hashicorp remote

  {
    "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "pulp_created": "2021-09-08T09:26:43.358984Z",
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "url": "https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": false,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2021-09-08T14:31:48.838823Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": [],
    "rate_limit": null,
    "sles_auth_token": null
  },

Created repository objects

Puppetlabs repository

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/",
    "pulp_created": "2021-09-08T12:38:31.425957Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/versions/1/",
    "name": "mirror-el8-x86_64-puppet-7",
    "description": null,
    "retained_versions": null,
    "remote": "/pulp/api/v3/remotes/rpm/rpm/f1fc36e9-bf54-48ab-b8e0-e93759c56dff/",
    "autopublish": true,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": "sha1",
    "package_checksum_type": "sha1",
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

Hashicorp repository

  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/",
    "pulp_created": "2021-09-08T09:26:43.420128Z",
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/",
    "pulp_labels": {},
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/versions/1/",
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "description": null,
    "retained_versions": null,
    "remote": "/pulp/api/v3/remotes/rpm/rpm/827d13b5-8d10-405c-9668-5b5cb1f970e9/",
    "autopublish": true,
    "metadata_signing_service": null,
    "retain_package_versions": 0,
    "metadata_checksum_type": "sha1",
    "package_checksum_type": "sha1",
    "gpgcheck": 0,
    "repo_gpgcheck": 0,
    "sqlite_metadata": false
  },

Distribution objects

Puppet distribution

  {
    "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/d8a10b0f-5a02-4605-820a-da357ae9ef6c/",
    "pulp_created": "2021-09-08T12:39:17.386142Z",
    "base_path": "rpm/pub/mirrors/el8/e86_64/puppet/7",
    "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/puppet/7/",
    "content_guard": null,
    "pulp_labels": {},
    "name": "mirror-el8-x86_64-puppet-7",
    "repository": "/pulp/api/v3/repositories/rpm/rpm/7f56a448-fcc2-472d-b41f-82283857b52d/",
    "publication": null
  },

Hashicorp distribution

  {
    "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/94a101ff-e4f5-4714-a752-732190b166fd/",
    "pulp_created": "2021-09-08T09:38:24.947840Z",
    "base_path": "rpm/pub/mirrors/el8/e86_64/hashicorp/stable",
    "base_url": "https://pulp.example.com/pulp/content/rpm/pub/mirrors/el8/e86_64/hashicorp/stable/",
    "content_guard": null,
    "pulp_labels": {},
    "name": "mirror-el8-x86_64-hashicorp-stable",
    "repository": "/pulp/api/v3/repositories/rpm/rpm/ca733f53-3b9e-4d3d-87cd-8abab43e8d04/",
    "publication": null
  },

History

#1 Updated by dalley 3 months ago

  • Triaged changed from No to Yes
  • Quarter set to Q4-2021

#2 Updated by dalley 3 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#3 Updated by dalley 2 months ago

  • Sprint set to Sprint 105

#4 Updated by rchan 2 months ago

  • Sprint changed from Sprint 105 to Sprint 106

#5 Updated by dalley 2 months ago

Cannot reproduce on master branch, will need to test again on 3.14.4

There were some semi-related changes that landed in master already and were backported to 3.14.4 so it is possible that this is already fixed.

#6 Updated by dalley 2 months ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

Unassigning this due to some other issues that need attention, will pick it back up eventually if we can confirm that it's still an issue (and what I'm missing, if so)

#7 Updated by dalley about 2 months ago

  • Priority changed from Normal to Low

#8 Updated by dalley about 2 months ago

  • Priority changed from Low to Normal
  • Sprint deleted (Sprint 106)

Please register to edit this issue

Also available in: Atom PDF