Issue #8713: Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist - RPM Support - Pulp

Actions

Send by e-mail Copy link

Issue #8713

closed

Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist

Added by ttereshc over 3 years ago. Updated about 3 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

ttereshc

Sprint/Milestone:

3.15.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Sprint 101

Quarter:

Description

To reproduce:

Pulp 3: sync/publish/distribute RHEL8 Appstream (which has a group "conflicts-appstream" which has an empty packagelist)
Pulp 2: sync from the repo above, observe the following error

 'NoneType' object has no attribute 'findall'
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 321, in run
     group.GROUP_TAG)
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 950, in get_comps_file_units
     self.save_fileless_units(group_file_handle, tag, process_func, mutable_type=True)
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 1006, in save_fileless_units
     for model in package_info_generator:
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
     package_info = process_func(element)
   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
     packagelist.findall('packagereq'))
 AttributeError: 'NoneType' object has no attribute 'findall'
INFO: [ac1cd020] Task failed : [ac1cd020-c06c-4d53-ac08-1adbab8c41b1]
 Task pulp.server.managers.repo.sync.sync[ac1cd020-c06c-4d53-ac08-1adbab8c41b1] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
     R = retval = fun(*args, **kwargs) 
   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
     return super(Task, self).__call__(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
     return super(PulpTask, self).__call__(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
     return self.run(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 860, in sync
     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
 PulpExecutionException: Importer indicated a failed response

Pulp 3 uses libcomps to generate a comps.xml file and it doesn't generate empty XML elements.
Pulp 2 expects packagelist to be always present, even if empty.
It also seems to be a mandatory however it is according to the unofficial spec.

Related issues

Actions

Copy link

Updated by ttereshc over 3 years ago

While we are figuring out where to fix it (hopefully in libcomps), here is patch to workaround the problem if you really need it.
It ignores empty groups, they likely do not bring much value, so as a workaround I think it's fine to skip them.

diff --git a/pulp_rpm/app/tasks/publishing.py b/pulp_rpm/app/tasks/publishing.py
index 1604a48..3528808 100644
--- a/pulp_rpm/app/tasks/publishing.py
+++ b/pulp_rpm/app/tasks/publishing.py
@@ -501,7 +501,7 @@ def create_repomd_xml(
 
     comps.toxml_f(
         comps_xml_path,
-        xml_options={"default_explicit": True, "empty_groups": True, "uservisible_explicit": True},
+        xml_options={"default_explicit": True, "empty_groups": False, "uservisible_explicit": True},
     )
 
     pri_xml.close()

Actions

Copy link

Updated by ttereshc over 3 years ago

There is an empty_packages=True option available in libcomps, see https://github.com/kontura/libcomps/commit/8b1705786433476a75bf4f4de3ca7240a1225388#diff-7adbf558c7e5bb29edf9dc76d821bf71d5698f8c775325d6ae8e5f7b5d6f45e2R88

Actions

Copy link

Updated by dalley over 3 years ago

Triaged changed from No to Yes
Sprint set to Sprint 97

Actions

Copy link

Updated by rchan over 3 years ago

Sprint changed from Sprint 97 to Sprint 98

Actions

Copy link

Updated by rchan over 3 years ago

Sprint changed from Sprint 98 to Sprint 99

Actions

Copy link

Updated by rchan over 3 years ago

Sprint changed from Sprint 99 to Sprint 100

Actions

Copy link

Updated by rchan over 3 years ago

Sprint changed from Sprint 100 to Sprint 101

Actions

Copy link

Updated by ttereshc over 3 years ago

Status changed from NEW to ASSIGNED
Assignee set to ttereshc

Actions

Copy link

#10

Updated by pulpbot over 3 years ago

Status changed from ASSIGNED to POST

PR: https://github.com/pulp/pulp_rpm/pull/2056

Added by ttereshc over 3 years ago

Revision 7673cab7 | View on GitHub

Configure createrepo_c to always generate packagelist element within a package group.

Pulp 2 relies on the packagelist element being present, even if it's empty. It also seems to be a mandatory element according to this unofficial spec https://pagure.io/rpm-metadata/blob/master/f/fedora/comps-schema/comps.dtd#_6

Before this fix, the packagelist element would only be generated if there is at least one package in a group.

closes #8713 https://pulp.plan.io/issues/8713

Actions

Copy link

#11