Issue #8713
closedPulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist
Description
To reproduce:
- Pulp 3: sync/publish/distribute RHEL8 Appstream (which has a group "conflicts-appstream" which has an empty packagelist)
- Pulp 2: sync from the repo above, observe the following error
'NoneType' object has no attribute 'findall'
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 321, in run
group.GROUP_TAG)
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 950, in get_comps_file_units
self.save_fileless_units(group_file_handle, tag, process_func, mutable_type=True)
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 1006, in save_fileless_units
for model in package_info_generator:
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
package_info = process_func(element)
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
packagelist.findall('packagereq'))
AttributeError: 'NoneType' object has no attribute 'findall'
INFO: [ac1cd020] Task failed : [ac1cd020-c06c-4d53-ac08-1adbab8c41b1]
Task pulp.server.managers.repo.sync.sync[ac1cd020-c06c-4d53-ac08-1adbab8c41b1] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
return super(Task, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
return super(PulpTask, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 860, in sync
raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
PulpExecutionException: Importer indicated a failed response
Pulp 3 uses libcomps to generate a comps.xml file and it doesn't generate empty XML elements.
Pulp 2 expects packagelist
to be always present, even if empty.
It also seems to be a mandatory however it is according to the unofficial spec.
Related issues
Updated by ttereshc over 3 years ago
While we are figuring out where to fix it (hopefully in libcomps), here is patch to workaround the problem if you really need it.
It ignores empty groups, they likely do not bring much value, so as a workaround I think it's fine to skip them.
diff --git a/pulp_rpm/app/tasks/publishing.py b/pulp_rpm/app/tasks/publishing.py
index 1604a48..3528808 100644
--- a/pulp_rpm/app/tasks/publishing.py
+++ b/pulp_rpm/app/tasks/publishing.py
@@ -501,7 +501,7 @@ def create_repomd_xml(
comps.toxml_f(
comps_xml_path,
- xml_options={"default_explicit": True, "empty_groups": True, "uservisible_explicit": True},
+ xml_options={"default_explicit": True, "empty_groups": False, "uservisible_explicit": True},
)
pri_xml.close()
Updated by ttereshc over 3 years ago
There is an empty_packages=True option available in libcomps, see https://github.com/kontura/libcomps/commit/8b1705786433476a75bf4f4de3ca7240a1225388#diff-7adbf558c7e5bb29edf9dc76d821bf71d5698f8c775325d6ae8e5f7b5d6f45e2R88
Updated by dalley over 3 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 97
Updated by ttereshc over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ttereshc
Updated by pulpbot over 3 years ago
- Status changed from ASSIGNED to POST
Added by ttereshc over 3 years ago
Updated by ttereshc over 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset 7673cab74e40223fcfb79ada792345356e5945f9.
Updated by dalley over 3 years ago
- Copied to Backport #9193: Backport #8713 "Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist" to 3.14.z added
Updated by ttereshc over 3 years ago
- Copied to Backport #9195: Backport #8713 "Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist" to 3.11.z added
Updated by pulpbot over 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Configure createrepo_c to always generate
packagelist
element within a package group.Pulp 2 relies on the packagelist element being present, even if it's empty. It also seems to be a mandatory element according to this unofficial spec https://pagure.io/rpm-metadata/blob/master/f/fedora/comps-schema/comps.dtd#_6
Before this fix, the packagelist element would only be generated if there is at least one package in a group.
closes #8713 https://pulp.plan.io/issues/8713