Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2622

closed

Sync fails when non-ASCII characters are present in primary.xml

Added by ttereshc over 7 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
ttereshc
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.12.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 16
Quarter:

Description

Before this change https://github.com/pulp/pulp_rpm/pull/1028, utf-8 decoding for primary.xml snippets happened inside the change_location_tag.
After that change decoding happens outside of it but only for upload case, so sync case has to be adjusted to this change or decoding should be moved back to change_location_tag.

To reproduce, create and sync EPEL repo:

pulp-admin rpm repo create --repo-id epel7 --relative-url epel7 --feed https://dl.fedoraproject.org/pub/epel/7/x86_64/ --download-policy on_demand
pulp-admin rpm repo sync  run --repo-id epel7

Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072) 'ascii' codec can't decode byte 0xc3 in position 884: ordinal not in range(128)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072) Traceback (most recent call last):
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 269, in run
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     self.update_content(metadata_files, url)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 549, in update_c
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     self.download_rpms(metadata_files, rpms_to_download, url)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 780, in download
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     unit = self.add_rpm_unit(metadata_files, unit)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 702, in add_rpm_
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     metadata_files.add_repodata(unit)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py", line 372, 
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     model.repodata['primary'] = change_location_tag(raw_xml, model.filename)
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)   File "/home/user1/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/rpm.py", line 75, in chan
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072)     return first_portion + location + end_portion
Mar 07 11:01:39 dev pulp[16378]: pulp_rpm.plugins.importers.yum.sync:ERROR: (16378-91072) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 884: ordinal not in range(128)
Mar 07 11:01:39 dev pulp[16378]: pulp.server.async.tasks:INFO: Task failed : [b8b85b93-dd63-487f-8b40-ce5384bdd9a8]
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072) Task pulp.server.managers.repo.sync.sync[b8b85b93-dd63-487f-8b40-ce5384bdd9a8] raised unexpected: PulpExecutionException('I
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072) Traceback (most recent call last):
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     R = retval = fun(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/home/user1/devel/pulp/server/pulp/server/async/tasks.py", line 506, in __call__
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     return super(Task, self).__call__(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/home/user1/devel/pulp/server/pulp/server/async/tasks.py", line 107, in __call__
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     return super(PulpTask, self).__call__(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     return self.run(*args, **kwargs)
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)   File "/home/user1/devel/pulp/server/pulp/server/controllers/repository.py", line 820, in sync
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Mar 07 11:01:39 dev pulp[16314]: celery.worker.job:ERROR: (16314-91072) PulpExecutionException: Importer indicated a failed response

Related issues

Related to RPM Support - Issue #1903: RPM import traceback (non-utf-8 metadata slipping through)CLOSED - CURRENTRELEASEjortel@redhat.comActions
Actions
Copy link
 #1

Updated by ttereshc over 7 years ago

  • Status changed from NEW to POST
  • Assignee set to ttereshc
Actions
Copy link
 #2

Updated by bizhang over 7 years ago

  • Sprint/Milestone set to 34
  • Triaged changed from No to Yes

Added by ttereshc over 7 years ago

Revision 2599e0d7 | View on GitHub

Fix sync for the case when primary.xml contains non-ASCII characters

closes #2622 https://pulp.plan.io/issues/2622

Actions
Copy link
 #3

Updated by ttereshc over 7 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #4

Updated by ttereshc over 7 years ago

  • Platform Release set to 2.12.2
Actions
Copy link
 #7

Updated by semyers over 7 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #8

Updated by pthomas@redhat.com over 7 years ago

The following passed.

[root@bkr-hv01-guest11 ~]# pulp-admin rpm repo create --repo-id epel7 --relative-url epel7 --feed https://dl.fedoraproject.org/pub/epel/7/x86_64/ --download-policy on_demand
Successfully created repository [epel7]

[root@bkr-hv01-guest11 ~]# pulp-admin rpm repo sync  run --repo-id epel7
+----------------------------------------------------------------------+
                    Synchronizing Repository [epel7]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[/]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       11339/11339 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[|]
... completed

Importing package groups/categories...
[|]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
11339 of 11339 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
3508 of 3508 items
... completed

Publishing Comps file
[==================================================] 100%
214 of 214 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[|]
... completed

Writing Listings File
[-]
... completed

Task Succeeded
Actions
Copy link
 #9

Updated by Ichimonji10 over 7 years ago

Preethi, should this issue be marked as "verified?"

Actions
Copy link
 #11

Updated by semyers over 7 years ago

We had a good team chat about the "Verification Required" flag on Monday, and decided that the release of 2.12.2 should not be blocked on the verification of this issue.

Actions
Copy link
 #12

Updated by bizhang over 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #13

Updated by ttereshc over 7 years ago

  • Related to Issue #1903: RPM import traceback (non-utf-8 metadata slipping through) added
Actions
Copy link
 #15

Updated by bmbouter over 6 years ago

  • Sprint set to Sprint 16
Actions
Copy link
 #16

Updated by bmbouter over 6 years ago

  • Sprint/Milestone deleted (34)
Actions
Copy link
 #17

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF