Project

Profile

Help

Issue #8700

CentOS 8 stream repositories fails to synchronize - 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte

Added by adam.tkac@gooddata.com 6 months ago. Updated 5 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Hello pulp upstream,

we are regularly mirroring various CentOS stream repos, but recently "sync" tasks started to fail with following traceback:

May  5 12:20:22 pulp32 rq[2083851]: Traceback (most recent call last):
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/rq/worker.py", line 1008, in perform_job
May  5 12:20:22 pulp32 rq[2083851]:    rv = job.perform()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/rq/job.py", line 706, in perform
May  5 12:20:22 pulp32 rq[2083851]:    self._result = self._execute()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/rq/job.py", line 729, in _execute
May  5 12:20:22 pulp32 rq[2083851]:    result = self.func(*self.args, **self.kwargs)
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 255, in synchronize
May  5 12:20:22 pulp32 rq[2083851]:    dv.create()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
May  5 12:20:22 pulp32 rq[2083851]:    loop.run_until_complete(pipeline)
May  5 12:20:22 pulp32 rq[2083851]:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
May  5 12:20:22 pulp32 rq[2083851]:    return future.result()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
May  5 12:20:22 pulp32 rq[2083851]:    await asyncio.gather(*futures)
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
May  5 12:20:22 pulp32 rq[2083851]:    await self.run()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 439, in run
May  5 12:20:22 pulp32 rq[2083851]:    await self.parse_modules_metadata()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 495, in parse_modules_metadata
May  5 12:20:22 pulp32 rq[2083851]:    modules_metadata_parser.parse()
May  5 12:20:22 pulp32 rq[2083851]:  File "/opt/gdc/pulp3/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 747, in parse
May  5 12:20:22 pulp32 rq[2083851]:    content = moduleyaml.read()
May  5 12:20:22 pulp32 rq[2083851]:  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
May  5 12:20:22 pulp32 rq[2083851]:    (result, consumed) = self._buffer_decode(data, self.errors, final)
May  5 12:20:22 pulp32 rq[2083851]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte

We are using following versions:

[root@pulp32:~] pulp status
{
  "versions": [
    {
      "component": "core",
      "version": "3.12.1"
    },
    {
      "component": "rpm",
      "version": "3.10.0"
    },
    {
      "component": "file",
      "version": "1.7.0"
    }

Remote configuration:

[root@pulp32:~] pulp rpm remote show --name c8s-baseos-remote
{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/dc58e56e-4a00-47f1-af0d-a9b6786af51c/",
  "pulp_created": "2021-05-05T02:44:02.345456Z",
  "name": "c8s-baseos-remote",
  "url": "http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "pulp_last_updated": "2021-05-05T02:44:02.345474Z",
  "download_concurrency": 10,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "sles_auth_token": null
}

How to reproduce:

[root@pulp32:~] pulp rpm repository sync --name c8s-baseos --remote c8s-baseos-remote
Started background task /pulp/api/v3/tasks/e50f608f-9bf7-48a9-ad6a-03efb34014d6/
.Error: Task /pulp/api/v3/tasks/e50f608f-9bf7-48a9-ad6a-03efb34014d6/ failed: ''utf-8' codec can't decode byte 0xfd in position 0: invalid start byte'

Please let me know if you need any more details. Thank you in advance!


Related issues

Related to RPM Support - Backport #8710: Backport Centos Stream metadata fix to 3.10.zCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Backport #8751: Backport #8700 to 3.11CLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 29009c83 View on GitHub
Added by Laura Hild 6 months ago

Allow .xz modules.yaml

closes: #8700 https://pulp.plan.io/issues/8700

Revision 2a0bfe65 View on GitHub
Added by dalley 6 months ago

Use libmodulemd to open module metadata files

Libmodulemd can auto-detect compression

re: #8700 https://pulp.plan.io/issues/8700

History

#1 Updated by iballou 6 months ago

I was able to sync BaseOS from this URL: http://repo.ialab.dsu.edu/centos/8-stream/BaseOS/x86_64/os/, but AppStream failed with the same error as above.

There is a Katello community post about this too: https://community.theforeman.org/t/foreman-2-4-katello-4-0-issue-syncing-repos-for-centos-stream/23455

#2 Updated by dalley 6 months ago

  • Project changed from Pulp to RPM Support

#3 Updated by dalley 6 months ago

  • Priority changed from Normal to High

#4 Updated by dalley 6 months ago

  • Triaged changed from No to Yes

#5 Updated by Anonymous 6 months ago

  • Status changed from NEW to MODIFIED

#6 Updated by dalley 6 months ago

  • Related to Backport #8710: Backport Centos Stream metadata fix to 3.10.z added

#7 Updated by dalley 6 months ago

Updated patch https://github.com/pulp/pulp_rpm/pull/1983/ (for backporting)

#8 Updated by ttereshc 5 months ago

#9 Updated by ttereshc 5 months ago

  • Sprint/Milestone set to 3.12.0

#10 Updated by pulpbot 5 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF