Issue #1768

Unable to sync RHEL 5 repositories with a distribution

Added by over 5 years ago. Updated over 2 years ago.

Start date:
Due date:
Estimated time:
3. High
Platform Release:
Sprint Candidate:
Pulp 2
Sprint 1


To reproduce:

  1. create a repo with a feed to or one of its mirrors (lazy or non-lazy, either are fine)
  2. Sync it

It fails to sync the distribution with:

pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128) ValidationError (Distribution:012649b5-c46a-4114-9f03-2948b070e360) (Field is required: ['variant'])
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128) Traceback (most recent call last):
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/", line 228, in run
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/", line 125, in run
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)     self._run(tmp_dir)
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/", line 179, in _run
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)   File "/usr/lib/python2.7/site-packages/mongoengine/", line 304, in save
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)     self.validate(clean=clean)
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)   File "/usr/lib/python2.7/site-packages/mongoengine/base/", line 413, in validate
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128)     raise ValidationError(message, errors=errors)
pulp_rpm.plugins.importers.yum.sync:ERROR: (9072-64128) ValidationError: ValidationError (Distribution:012649b5-c46a-4114-9f03-2948b070e360) (Field is required: ['variant'])

The reason for this is RHEL5 uses an older format for its treeinfo files, but our model does not handle it gracefully. The current format for these treeinfo files can be found here:

We need to make our distribution tree model more flexible and we need to carefully document which bits are there for backwards compatibility with RHEL 5. Furthermore, CentOS 7 doesn't seem to do what RHEL7 does with its treeinfo file.

upgrade_validation-20160311164238.log (79.2 KB) upgrade_validation-20160311164238.log mhrivnak, 03/11/2016 05:45 PM
treeinfos.tar.gz (20 KB) treeinfos.tar.gz, 04/04/2016 08:02 PM

Related issues

Related to RPM Support - Story #1769: As a user, I would like to be able to sync distribution treesCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 2e65b095 View on GitHub
Added by Jeremy Cline over 5 years ago

The Distribution model no longer requires variant

This is very much a short-term workaround. The model has many problems and should be fixed in the near future.

closes #1768


#1 Updated by mhrivnak over 5 years ago

I ran the 2.8 upgrade test on a mongodb dump from a pulp 2.6, which had sync'd the rhel 5.10 kickstart distribution. The same validation error was seen by the script, so this is a problem during model hydration.

#2 Updated by over 5 years ago

This is not a problem on model hydration. I'm not familiar enough with the internals of the validation on hydration/save to speak for why this is the case, but the problem is that we cannot ever save any unit with a null value in a field MongoEngine has declared as ``required`` (from my brief investigation).

I think that this will not have a huge effect on users (they can still do most activities, just not sync RHEL5 distributions) so we can push this to a z release without too much damage. However, this is very likely the tip of the iceburg when it comes to data validation on hydration and on save. We modify documents outside of MongoEngine all over the place and if any of those fail to match the validation requires of our MongoEngine models it will crash and burn when MongoEngine attempts to hydrate them, or when it attempts to save them.

I know we have that script to hydrate and save, but it clearly hasn't even been used on all repositories synced from the CDN, let alone other repositories out there.

#3 Updated by bmbouter over 5 years ago

+1 to moving this to 2.8.1

Regarding the validation and hydration issues we need to make sure that the test scripts are identified in the release notes, upgrade notes, and probably some proactive mail on pulp-list.

#4 Updated by mhrivnak over 5 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.1
  • Triaged changed from No to Yes

#5 Updated by pcreech over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to pcreech

#6 Updated by pcreech over 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (pcreech)

#7 Updated by semyers over 5 years ago

  • Platform Release changed from 2.8.1 to 2.8.2

#8 Updated by mhrivnak over 5 years ago

  • Sprint/Milestone set to 19

#10 Updated by over 5 years ago

  • Related to Story #1769: As a user, I would like to be able to sync distribution trees added

#11 Updated by over 5 years ago

I've attached a range of treeinfo files for whoever picks this up to look at. At least some of the work for #1769 will need to be done for this issue as well, so I have made them related.

#13 Updated by over 5 years ago

  • Description updated (diff)

I've spent a lot of time looking into this and thinking about what we could do. The real issue is that our model is just wrong. We might be able to work around this specific problem by defaulting the variant (and any other fields the RHEL 5 treeinfo file doesn't have that our model requires) to an empty string or something. If won't fix all our problems (we don't sync addons, we don't checksum files, we yank out treeinfo fields, etc), but it might be the best we can do without scrapping the model (and modifying a lot of our distribution sync code) and doing it right.

#14 Updated by mhrivnak over 5 years ago

It appears that pulp < 2.8 stores the "variant" as null in the database for a case like rhel 5.10, where there is no variant. Mongoengine seems to not support the concept of having a null value in the database at all, but rather removes or omits the key when its value is None.

Related discussion:

This whole model definitely deserves a re-think. But to address this immediate bug, we might be able to make a simple custom field that gets us back on the road.

#15 Updated by semyers over 5 years ago

  • Platform Release changed from 2.8.2 to 2.8.3

#16 Updated by over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to

#17 Updated by over 5 years ago

  • Status changed from ASSIGNED to POST

#18 Updated by Anonymous over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#20 Updated by over 5 years ago

[root@qe-blade-03 ~]# rpm -qa |grep pulp-server

[root@pulp ~]# pulp-admin rpm repo create --feed --feed-ca-cert cdn-ca.crt --feed-cert cdn.crt --feed-key cdn.key --repo-id rhel511-ks
Successfully created repository [rhel511-ks]

[root@pulp ~]#
[root@pulp ~]#
[root@pulp ~]# pulp-admin rpm repo sync run --repo-id rhel511-ks
Synchronizing Repository [rhel511-ks]

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
... completed

Downloading repository content...
[==================================================] 100%
RPMs: 3372/3372 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[================================================= ] 98%
Distributions: 107/109 items

[==================================================] 100%
Distributions: 109/109 items
... completed

Importing errata...
... completed

Importing package groups/categories...
... completed

Cleaning duplicate packages...
... completed

Task Succeeded

Initializing repo metadata
... completed

Publishing Distribution files
... completed

Publishing RPMs
[==================================================] 100%
3372 of 3372 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
... completed

Publishing Comps file
... completed

Publishing Metadata.
... completed

Closing repo metadata
... completed

Generating sqlite files
... skipped

Publishing files to web
... completed

Writing Listings File
... completed

Task Succeeded

#21 Updated by semyers over 5 years ago

  • Status changed from MODIFIED to 5

#22 Updated by over 5 years ago

  • Status changed from 5 to 6

Moving to verified as per comment #20

#24 Updated by semyers over 5 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#26 Updated by bmbouter over 3 years ago

  • Sprint set to Sprint 1

#27 Updated by bmbouter over 3 years ago

  • Sprint/Milestone deleted (19)

#28 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF