Project

Profile

Help

Issue #5131

Mongoengine has a bug which prevents .only() from working in combination with .as_pymongo()

Added by dalley 4 months ago. Updated about 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Low
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.21.0
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 56

Description

The recursive depsolving code needs to load the contents of an entire repository into libsolv, and it needs to do so quickly. At this scale, the overhead of creating a Python model object for every single content unit becomes significant, so instead of dump the data into Python dictionaries using .as_pymongo(). Additionally, we only want to load the subset of the field data that we need for the depsolving process, so we use .only() to select only the fields we want to load, to reduce memory and database load.

The version of mongoengine currently being used is mongoengine 0.10.5, released in late 2015. It has a bug wherein using .only() in combination with .as_pymongo() in combination with nested fields on the model (e.g. a ListField) returns a list containing a single empty dictionary instead of the data it was supposed to return.

The net result of that is that we cannot load that data properly using the current version of libsolv and the .only() method with the version of mongoengine currently being shipped. The workaround, which was heretofore undocumented, was to instead use .excludes() to blacklist fields we don't want instead of .only() to whitelist the fields we do want. I removed this workaround accidentally during a refactor but will re-introduce it after discovering this problem.

The regression went unnoticed because the version of mongoengine provided by Pulp is overridden by a newer version in our Fedora 28 dev environment, which does not contain this defect. Only CentOS and RHEL installations demonstrate it.

Associated revisions

Revision fc81a127 View on GitHub
Added by dalley 4 months ago

Re-introduce workaround for mongoengine bug

The version of mongoengine we're using is broken, and I accidentally
removed a workaround. Re-introduce the workaround and document it as
such.

re: #5131
https://pulp.plan.io/issues/5131

History

#1 Updated by dalley 4 months ago

  • Description updated (diff)

#2 Updated by dalley 4 months ago

  • Project changed from Packaging to RPM Support
  • Status changed from NEW to POST
  • Assignee set to dalley

#3 Updated by dalley 4 months ago

  • Status changed from POST to MODIFIED
  • Sprint set to Sprint 56

Workaround put in place, no need to upgrade unless we hit other serious issues.

#4 Updated by dalley 3 months ago

  • Platform Release set to 2.21.0

#5 Updated by dalley about 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF