Pulp

Known performance issues.
https://github.com/MongoEngine/mongoengine/issues/1230
https://github.com/MongoEngine/mongoengine/issues/1137
https://github.com/MongoEngine/mongoengine/issues/1073

Profiler showing most time spent in Document.__init__() called by Document._from_son() for a test fetching unit associations using pymongo then calling Document._from_son() for each record. Also, this test used a bare bones mongoengine Document subclass instead of the one in the pulp model. The test is intended to isolate the hydration and be free of pulp modeling side effects.

Here we see the profiler statistics if a call to repository.find_repo_content_units(r, yield_content_unit=True).

I did a little related digging. Instantiating RPM model instances, nearly half the time is spent in the to_python() method of ComplexBaseField, which is the parent class for ListField and DictField. Long-term, minimizing the use of those fields could help.

In the mean time, since we probably know the data types to expect inside those structures, we could make our own ListField and DictField subclasses with a custom to_python() that only does the minimum required for the expected data in that specific field. The generic implementation of to_python() does a ton of weak inspection via hasattr, isinstance, try...except with various type-specific operations, and other weird uses of intermediate data structures. Making our own might save a substantial amount of time.

For this query with about 9100 RPMS:

    for x in RPM.objects.filter().only(*RPM.unit_key_fields):
        pass

I got the time down from 9.6s to 6.8s just with this patch to mongoengine:

diff --git a/mongoengine/base/document.py b/mongoengine/base/document.py
index 6353162..f6006be 100644
--- a/mongoengine/base/document.py
+++ b/mongoengine/base/document.py
@@ -91,8 +91,8 @@ class BaseDocument(object):
         for key, field in self._fields.iteritems():
             if self._db_field_map.get(key, key) in __only_fields:
                 continue
-            value = getattr(self, key, None)
-            setattr(self, key, value)
+            if not hasattr(self, key):
+                setattr(self, key, None)

         if "_cls" not in values:
             self._cls = self._class_name

Before running too far with that, we need to ensure there wasn't some reason for mongoengine doing the otherwise strange-looking getattr followed by setattr. But this might be a low-hanging-fruit improvement.

Before we get too excited, mongoengine's unit test suite isn't very fond of my little optimization. I'll have to dig into why.

FAILED (failures=23, errors=31, skipped=4)

I did some testing of pulp 2.6.5, 2.7.1, and 2.8.1-rc. I created a centos7 VM with 2.6.5 and sync'd a base rhel7 repo, with 4620 packages. I then created two copies of the VM, and upgraded them to 2.7.1 and 2.8.1-rc respectively.

Hardware was a 4-year-old desktop machine with a single spinning disk that's rather slow.

On each VM I copied all rpms from the rhel7 repo into a new repo. I did this three times and calculated the average time:

2.6.5: 16.17s
2.7.1: 20.35s
2.8.1: 34.24s

Since I was doing orphan purges between tests anyway, I gathered times on that operation also:

2.6.5:  2.47s
2.7.1:  2.84s
2.8.1: 10.15s

The big jump on 2.8.x is mostly due to using mongoengine. We've discovered that instantiating mongoengine classes is a surprisingly expensive operation. Part of the copy performance jump also may be due to new behavior that guards against duplicate NEVRA in a repo, but I am only speculating on this: https://pulp.plan.io/issues/1461

I don't know why 2.7.1 is 25% slower than 2.6.5.