Project

Profile

Help

Issue #2622

Sync fails when non-ASCII characters are present in primary.xml

Added by ttereshc 6 months ago. Updated 6 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Severity:
2. Medium
Version:
Platform Release:
2.12.2
Blocks Release:
2.12.z, 2.13.z
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
No

Description

Before this change https://github.com/pulp/pulp_rpm/pull/1028, utf-8 decoding for primary.xml snippets happened inside the change_location_tag.
After that change decoding happens outside of it but only for upload case, so sync case has to be adjusted to this change or decoding should be moved back to change_location_tag.

To reproduce, create and sync EPEL repo:

pulp-admin rpm repo create --repo-id epel7 --relative-url epel7 --feed https://dl.fedoraproject.org/pub/epel/7/x86_64/ --download-policy on_demand
pulp-admin rpm repo sync  run --repo-id epel7

Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072) 'ascii' codec can't decode byte 0xc3 in position 884: ordinal not in range(128)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072) Traceback (most recent call last):
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 269, in run
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     self.update_content(metadata_files, url)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 549, in update_c
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     self.download_rpms(metadata_files, rpms_to_download, url)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 780, in download
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     unit = self.add_rpm_unit(metadata_files, unit)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 702, in add_rpm_
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     metadata_files.add_repodata(unit)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py", line 372, 
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     model.repodata['primary'] = change_location_tag(raw_xml, model.filename)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/rpm.py", line 75, in chan
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     return first_portion + location + end_portion
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 884: ordinal not in range(128)
Mar 07 11:01:39 dev pulp[16378]: pulp.server.async.tasks:INFO: Task failed : [b8b85b93-dd63-487f-8b40-ce5384bdd9a8]
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072) Task pulp.server.managers.repo.sync.sync[b8b85b93-dd63-487f-8b40-ce5384bdd9a8] raised unexpected: PulpExecutionException('I
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072) Traceback (most recent call last):
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     R = retval = fun(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/home/user1/devel/pulp/server/pulp/server/async/tasks.py", line 506, in __call__
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     return super(Task, self).__call__(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/home/user1/devel/pulp/server/pulp/server/async/tasks.py", line 107, in __call__
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     return super(PulpTask, self).__call__(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     return self.run(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/home/user1/devel/pulp/server/pulp/server/controllers/repository.py", line 820, in sync
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072) PulpExecutionException: Importer indicated a failed response

Related issues

Related to RPM Support - Issue #1903: RPM import traceback (non-utf-8 metadata slipping through) CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision 2599e0d7 View on GitHub
Added by ttereshc 6 months ago

Fix sync for the case when primary.xml contains non-ASCII characters

closes #2622
https://pulp.plan.io/issues/2622

History

#1 Updated by ttereshc 6 months ago

  • Status changed from NEW to POST
  • Assignee set to ttereshc

#2 Updated by bizhang 6 months ago

  • Sprint/Milestone set to Sprint 16
  • Triaged changed from No to Yes

#3 Updated by ttereshc 6 months ago

  • Status changed from POST to MODIFIED

#4 Updated by ttereshc 6 months ago

  • Platform Release set to 2.12.2

#5 Updated by pthomas@redhat.com 5 months ago

  • Smash Test set to 581

#6 Updated by semyers 5 months ago

  • Verification Required changed from No to Yes

#7 Updated by semyers 5 months ago

  • Status changed from MODIFIED to ON_QA

#8 Updated by pthomas@redhat.com 5 months ago

The following passed.

[root@bkr-hv01-guest11 ~]# pulp-admin rpm repo create --repo-id epel7 --relative-url epel7 --feed https://dl.fedoraproject.org/pub/epel/7/x86_64/ --download-policy on_demand
Successfully created repository [epel7]

[root@bkr-hv01-guest11 ~]# pulp-admin rpm repo sync  run --repo-id epel7
+----------------------------------------------------------------------+
                    Synchronizing Repository [epel7]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[/]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       11339/11339 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[|]
... completed

Importing package groups/categories...
[|]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
11339 of 11339 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
3508 of 3508 items
... completed

Publishing Comps file
[==================================================] 100%
214 of 214 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[|]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

#9 Updated by Ichimonji10 5 months ago

Preethi, should this issue be marked as "verified?"

#10 Updated by pthomas@redhat.com 5 months ago

  • Verified changed from No to Yes

#11 Updated by semyers 5 months ago

  • Verification Required changed from Yes to No

We had a good team chat about the "Verification Required" flag on Monday, and decided that the release of 2.12.2 should not be blocked on the verification of this issue.

#12 Updated by bizhang 4 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#13 Updated by ttereshc 7 days ago

  • Related to Issue #1903: RPM import traceback (non-utf-8 metadata slipping through) added

Please register to edit this issue

Also available in: Atom PDF