Project

Profile

Help

Issue #8713

closed

Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist

Added by ttereshc over 3 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 101
Quarter:

Description

To reproduce:

  • Pulp 3: sync/publish/distribute RHEL8 Appstream (which has a group "conflicts-appstream" which has an empty packagelist)
  • Pulp 2: sync from the repo above, observe the following error
 'NoneType' object has no attribute 'findall'
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 321, in run
     group.GROUP_TAG)
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 950, in get_comps_file_units
     self.save_fileless_units(group_file_handle, tag, process_func, mutable_type=True)
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 1006, in save_fileless_units
     for model in package_info_generator:
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
     package_info = process_func(element)
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
     packagelist.findall('packagereq'))
 AttributeError: 'NoneType' object has no attribute 'findall'
INFO: [ac1cd020] Task failed : [ac1cd020-c06c-4d53-ac08-1adbab8c41b1]
 Task pulp.server.managers.repo.sync.sync[ac1cd020-c06c-4d53-ac08-1adbab8c41b1] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
     R = retval = fun(*args, **kwargs) 
   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
     return super(Task, self).__call__(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
     return super(PulpTask, self).__call__(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
     return self.run(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 860, in sync
     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
 PulpExecutionException: Importer indicated a failed response

Pulp 3 uses libcomps to generate a comps.xml file and it doesn't generate empty XML elements.
Pulp 2 expects packagelist to be always present, even if empty.
It also seems to be a mandatory however it is according to the unofficial spec.


Related issues

Copied to RPM Support - Backport #9193: Backport #8713 "Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Copied to RPM Support - Backport #9195: Backport #8713 "Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist" to 3.11.zCLOSED - CURRENTRELEASEttereshc

Actions
Actions #1

Updated by ttereshc over 3 years ago

While we are figuring out where to fix it (hopefully in libcomps), here is patch to workaround the problem if you really need it.
It ignores empty groups, they likely do not bring much value, so as a workaround I think it's fine to skip them.

diff --git a/pulp_rpm/app/tasks/publishing.py b/pulp_rpm/app/tasks/publishing.py
index 1604a48..3528808 100644
--- a/pulp_rpm/app/tasks/publishing.py
+++ b/pulp_rpm/app/tasks/publishing.py
@@ -501,7 +501,7 @@ def create_repomd_xml(
 
     comps.toxml_f(
         comps_xml_path,
-        xml_options={"default_explicit": True, "empty_groups": True, "uservisible_explicit": True},
+        xml_options={"default_explicit": True, "empty_groups": False, "uservisible_explicit": True},
     )
 
     pri_xml.close()


Actions #4

Updated by dalley over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 97
Actions #5

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #6

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #7

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #8

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #9

Updated by ttereshc over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
Actions #10

Updated by pulpbot over 3 years ago

  • Status changed from ASSIGNED to POST

Added by ttereshc over 3 years ago

Revision 7673cab7 | View on GitHub

Configure createrepo_c to always generate packagelist element within a package group.

Pulp 2 relies on the packagelist element being present, even if it's empty. It also seems to be a mandatory element according to this unofficial spec https://pagure.io/rpm-metadata/blob/master/f/fedora/comps-schema/comps.dtd#_6

Before this fix, the packagelist element would only be generated if there is at least one package in a group.

closes #8713 https://pulp.plan.io/issues/8713

Actions #11

Updated by ttereshc over 3 years ago

  • Status changed from POST to MODIFIED
Actions #12

Updated by dalley over 3 years ago

  • Copied to Backport #9193: Backport #8713 "Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist" to 3.14.z added
Actions #13

Updated by dalley over 3 years ago

  • Sprint/Milestone set to 3.15.0
Actions #15

Updated by ttereshc over 3 years ago

  • Copied to Backport #9195: Backport #8713 "Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist" to 3.11.z added
Actions #16

Updated by pulpbot over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF