Project

Profile

Help

Issue #5131

closed

Mongoengine has a bug which prevents .only() from working in combination with .as_pymongo()

Added by dalley over 4 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Low
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.21.0
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 56
Quarter:

Description

The recursive depsolving code needs to load the contents of an entire repository into libsolv, and it needs to do so quickly. At this scale, the overhead of creating a Python model object for every single content unit becomes significant, so instead of dump the data into Python dictionaries using .as_pymongo(). Additionally, we only want to load the subset of the field data that we need for the depsolving process, so we use .only() to select only the fields we want to load, to reduce memory and database load.

The version of mongoengine currently being used is mongoengine 0.10.5, released in late 2015. It has a bug wherein using .only() in combination with .as_pymongo() in combination with nested fields on the model (e.g. a ListField) returns a list containing a single empty dictionary instead of the data it was supposed to return.

The net result of that is that we cannot load that data properly using the current version of libsolv and the .only() method with the version of mongoengine currently being shipped. The workaround, which was heretofore undocumented, was to instead use .excludes() to blacklist fields we don't want instead of .only() to whitelist the fields we do want. I removed this workaround accidentally during a refactor but will re-introduce it after discovering this problem.

The regression went unnoticed because the version of mongoengine provided by Pulp is overridden by a newer version in our Fedora 28 dev environment, which does not contain this defect. Only CentOS and RHEL installations demonstrate it.

Actions #1

Updated by dalley over 4 years ago

  • Description updated (diff)

Added by dalley over 4 years ago

Revision fc81a127 | View on GitHub

Re-introduce workaround for mongoengine bug

The version of mongoengine we're using is broken, and I accidentally removed a workaround. Re-introduce the workaround and document it as such.

re: #5131 https://pulp.plan.io/issues/5131

Actions #2

Updated by dalley over 4 years ago

  • Project changed from Packaging to RPM Support
  • Status changed from NEW to POST
  • Assignee set to dalley
Actions #3

Updated by dalley over 4 years ago

  • Status changed from POST to MODIFIED
  • Sprint set to Sprint 56

Workaround put in place, no need to upgrade unless we hit other serious issues.

Actions #4

Updated by dalley over 4 years ago

  • Platform Release set to 2.21.0
Actions #5

Updated by dalley over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF