Issue #9133: RPM repository sync error - RPM Support - Pulp

Actions

Send by e-mail Copy link

Issue #9133

closed

RPM repository sync error

Added by gvde over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - NOTABUG

Priority:

High

Assignee:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Sprint 101

Quarter:

Description

I have been collecting all information on this problem here: https://community.theforeman.org/t/centos-8-4-baseos-sync-error

In short: occasionally my sync of the CentOS 8 BaseOS repository from centos.org ends with errors:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 22 Jul 2021 20:36:46 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"ded10b33-c063-4471-9182-3b62facbd36b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 22 Jul 2021 20:36:47 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"ded10b33-c063-4471-9182-3b62facbd36b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/ not found for repositoryversion."]

I can access the mentioned publication URI via API:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.dkrz.de/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/' | python -m json.tool
{
    "gpgcheck": 0,
    "metadata_checksum_type": "unknown",
    "package_checksum_type": "unknown",
    "pulp_created": "2021-07-22T20:36:04.962028Z",
    "pulp_href": "/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/",
    "repo_gpgcheck": 1,
    "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
    "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/4/",
    "sqlite_metadata": true
}

However, the repository for that is not found:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.dkrz.de/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/' | python -m json.tool
{
    "detail": "Not found."
}

Checking the pulpcore database I can find both:

pulpcore=# select * from rpm_rpmpublication where publication_ptr_id = '1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d';
          publication_ptr_id          | metadata_checksum_type | package_checksum_type | gpgcheck | repo_gpgcheck | sqlite_metadata 
--------------------------------------+------------------------+-----------------------+----------+---------------+-----------------
 1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d | unknown                | unknown               |        0 |             1 | t
(1 row)

pulpcore=# select * from rpm_rpmrepository where repository_ptr_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
          repository_ptr_id           | sub_repo | metadata_signing_service_id |         last_sync_remote_id          | last_sync_repo_version | last_sync_revision
_number |                                                                                        original_checksum_types                                           
                                             | retain_package_versions |                    last_sync_repomd_checksum                     | autopublish | gpgcheck 
| metadata_checksum_type | package_checksum_type | repo_gpgcheck | sqlite_metadata 
--------------------------------------+----------+-----------------------------+--------------------------------------+------------------------+-------------------
--------+----------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------+-------------------------+------------------------------------------------------------------+-------------+----------
+------------------------+-----------------------+---------------+-----------------
 3f9dc526-a51c-4a25-9547-95f82bedb3ee | t        |                             | 69abed5c-b1e1-452a-b58b-f0cc23548924 |                      4 | 8.4.2105          
        | {"group": "sha256", "other": "sha256", "modules": "sha256", "primary": "sha256", "group_xz": "sha256", "other_db": "sha256", "filelists": "sha256", "prim
ary_db": "sha256", "filelists_db": "sha256"} |                       0 | c21c5d2410544fccf2dcc78ce0f472fd8ed9d8c3fc0de1a8f8061a72629a5c7e | f           |        0 
| sha256                 | sha256                |             0 | f
(1 row)

Actions

Copy link

Updated by gvde over 3 years ago

I have meant to mention that it's on a Katello server with 4.1.1, i.e. pulpcore 3.14. Installed RPMs *pulp*:

pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulp-2to3-migration-0.12.0-1.el7.noarch
python3-pulp-ansible-0.8.0-1.el7.noarch
python3-pulp-certguard-1.4.0-1.el7.noarch
python3-pulp-container-2.7.0-1.el7.noarch
python3-pulpcore-3.14.1-1.el7.noarch
python3-pulp-deb-2.13.0-1.el7.noarch
python3-pulp-file-1.8.1-1.el7.noarch
python3-pulp-rpm-3.13.3-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.8.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.4.0-1.el7.noarch
tfm-rubygem-pulp_container_client-2.7.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.14.1-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.13.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.8.1-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.13.3-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el7.noarch

Actions

Copy link

Updated by gvde over 3 years ago

There seems to be something worse: I have just noticed that 5 days ago the sync for the baseos of CentOS 8 Stream (not 8.4 as above) ended with a warning, and again this morning. I have the 8 Stream repos only for testing, i.e. currently I only have one client and it's using the Library environment.

5 days ago:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 19 Jul 2021 02:00:55 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"b05492c2-dc3a-4cfb-9099-22674dcc652d", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 19 Jul 2021 02:00:57 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"b05492c2-dc3a-4cfb-9099-22674dcc652d", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/8c78f8f7-4278-44d7-bef9-30ce30843c4c/ not found for repositoryversion."]

Now that publication 8c78f8f7-4278-44d7-bef9-30ce30843c4c really doesn't exist. It's not found via API nor it is in the core_publication or rpm_rpmpublication tables.

This morning:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:40 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:41 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:41 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/ad1067e2-4255-4898-b55e-d2341e59cb99/ not found for repositoryversion."]

Now that publication ad1067e2-4255-4898-b55e-d2341e59 has the same issue as the baseos for 8.4 above: the publication references a repository which exists in the database but the API says "Not found".

What's worse: the client cannot access the repository anymore:

# dnf check-update
...
Errors during downloading metadata for repository 'ORG_centos8stream_BaseOS_x86_64_os':
  - Status code: 404 for https://foreman.example.com/pulp/repos/ORG/Library/custom/centos8stream/BaseOS_x86_64_os/repodata/repomd.xml (IP: xxxx:xxxx::1)
Error: Failed to download metadata for repo 'ORG_centos8stream_BaseOS_x86_64_os': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Actions

Copy link