Pulp

In addition, the sync failed after about 2.5 hours with the following:

"description": "stack depth limit exceeded\nHINT:  Increase the configuration parameter \"max_stack_depth\" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.\n", 
    "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 799, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 600, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 606, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/nfs/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in synchronize\n    dv.create()\n  File \"/home/vagrant/devel/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\n    return future.result()\n  File \"/home/vagrant/devel/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/home/vagrant/devel/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/home/vagrant/devel/pulpcore-plugin/pulpcore/plugin/stages/association_stages.py\", line 130, in run\n    self.new_version.remove_content(queryset_to_unassociate)\n  File \"/home/vagrant/nfs/pulp/pulpcore/app/models/repository.py\", line 399, in remove_content\n    q_set.update(version_removed=self)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 693, in update\n    rows = query.get_compiler(self.db).execute_sql(CURSOR)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1383, in execute_sql\n    cursor = super().execute_sql(result_type)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1065, in execute_sql\n    cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 68, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 77, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 85, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 85, in _execute\n    return self.cursor.execute(sql, params)\n"
}, 

Looking at the models.py[0], I don't see any indexes defined for the fields being used in that query. We should probably update the models with indexes for each individual field and the combination of them all together.

[0] https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models.py

I added a bunch of indexes and didn't notice much improvement:

"foobar" btree (name, epoch, version, release, arch, checksum_type, "pkgId", content_ptr_id)

should fit the query, but IANADBA

In addition, it appears that the query seems to increase by about 8K characters at every iteration:

1. tail /var/lib/pgsql/data/pg_log/postgresql-Mon.log | awk '{ print length }'
   1052291
   1061237
   1069792
   1078413
   1086990
   1095609
   1104472
   1113154
   1121694
   1130256

"unique_together" creates an index, so the indexes should already be there.

Here's a reformatted version of that query

UPDATE "pulp_app_repositorycontent" SET "version_removed_id" = 2 
WHERE ("pulp_app_repositorycontent"."content_id" IN 
    (SELECT U0."content_ptr_id" FROM "rpm_package" U0 WHERE (
        (U0."arch" = 'x86_64' AND U0."epoch" = '0' AND U0."name" = '0ad' AND U0."release" = '5.fc28' AND U0."version" = '0.0.22' AND NOT (U0."content_ptr_id" = 51)) OR
        (U0."arch" = 'noarch' AND U0."epoch" = '0' AND U0."name" = '0ad-data' AND U0."release" = '2.fc28' AND U0."version" = '0.0.22' AND NOT (U0."content_ptr_id" = 52)) OR 
        (U0."arch" = 'x86_64' AND U0."epoch" = '0' AND U0."name" = '0install' AND U0."release" = '1.fc27' AND U0."version" = '2.12.1' AND NOT (U0."content_ptr_id" = 53)) OR
        (U0."arch" = 'x86_64' AND U0."epoch" = '0' AND U0."name" = '0xFFFF' AND U0."release" = '15.fc26' AND U0."version" = '0.3.9' AND NOT (U0."content_ptr_id" = 54)) OR
        (U0."arch" = 'x86_64' AND U0."epoch" = '0' AND U0."name" = '2048-cli' AND U0."release" = '5.fc28' AND U0."version" = '0.9.1' AND NOT