Pulp

Round 1:

100k: 8 minutes 57 seconds

1k: 0 minutes 7 seconds

%timeit {c['_type']: c['count'] for c in repo_version.content.values('_type').annotate(count=Count('_type'))}

129 ms ± 855 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit list(Content.objects.filter(pk__in=repo_version.content))

1.14 s ± 8.28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Round 2 (101,000 pre-existing units):

100k: 13 minutes 21 seconds

1k: 0 minutes 11 seconds

%timeit {c['_type']: c['count'] for c in repo_version.content.values('_type').annotate(count=Count('_type'))}

235 ms ± 2.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit list(Content.objects.filter(pk__in=repo_version.content))

2.34 s ± 21 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Round 3 (202,000 pre-existing units):

100k: 26 minutes 13 seconds

1k: 0 minutes 23 seconds

%timeit {c['_type']: c['count'] for c in repo_version.content.values('_type').annotate(count=Count('_type'))}

721 ms ± 81.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit list(Content.objects.filter(pk__in=repo_version.content))

3.5 s ± 19.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Round 4 (303,000 pre-existing units):

100k: 38 minutes 46 seconds

1k: 0 minutes 28 seconds

%timeit {c['_type']: c['count'] for c in repo_version.content.values('_type').annotate(count=Count('_type'))}

937 ms ± 95.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit list(Content.objects.filter(pk__in=repo_version.content))

4.65 s ± 26.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Round 5 (fresh repo, background 404,000 pre-existing units):

100k: 18 minutes 31 seconds

1k: 0 minutes 18 seconds

%timeit {c['_type']: c['count'] for c in repo_version.content.values('_type').annotate(count=Count('_type'))}

280 ms ± 5.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit list(Content.objects.filter(pk__in=repo_version.content))

1.86 s ± 14.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Final database size: 736 MB

List of relations

Schema | Name | Type | Owner | Size | Description

--------+-----------------------------------+----------+-------+------------+-------------

public | auth_group | table | pulp | 0 bytes |

public | auth_group_id_seq | sequence | pulp | 8192 bytes |

public | auth_group_permissions | table | pulp | 0 bytes |

public | auth_group_permissions_id_seq | sequence | pulp | 8192 bytes |

public | auth_permission | table | pulp | 40 kB |

public | auth_permission_id_seq | sequence | pulp | 8192 bytes |

public | auth_user | table | pulp | 16 kB |

public | auth_user_groups | table | pulp | 0 bytes |

public | auth_user_groups_id_seq | sequence | pulp | 8192 bytes |

public | auth_user_id_seq | sequence | pulp | 8192 bytes |

public | auth_user_user_permissions | table | pulp | 0 bytes |

public | auth_user_user_permissions_id_seq | sequence | pulp | 8192 bytes |

public | django_admin_log | table | pulp | 8192 bytes |

public | django_admin_log_id_seq | sequence | pulp | 8192 bytes |

public | django_content_type | table | pulp | 8192 bytes |

public | django_content_type_id_seq | sequence | pulp | 8192 bytes |

public | django_migrations | table | pulp | 16 kB |

public | django_migrations_id_seq | sequence | pulp | 8192 bytes |

public | django_session | table | pulp | 8192 bytes |

public | file_filecontent | table | pulp | 56 MB |

public | file_filepublisher | table | pulp | 8192 bytes |

public | file_fileremote | table | pulp | 8192 bytes |

public | pulp_app_artifact | table | pulp | 8192 bytes |

public | pulp_app_content | table | pulp | 37 MB |

public | pulp_app_contentartifact | table | pulp | 41 MB |

public | pulp_app_contentguard | table | pulp | 8192 bytes |

public | pulp_app_createdresource | table | pulp | 8192 bytes |

public | pulp_app_distribution | table | pulp | 8192 bytes |

public | pulp_app_exporter | table | pulp | 8192 bytes |

public | pulp_app_progressreport | table | pulp | 56 kB |

public | pulp_app_publication | table | pulp | 0 bytes |

public | pulp_app_publishedartifact | table | pulp | 0 bytes |

public | pulp_app_publishedmetadata | table | pulp | 8192 bytes |

public | pulp_app_publisher | table | pulp | 8192 bytes |

public | pulp_app_remote | table | pulp | 16 kB |

public | pulp_app_remoteartifact | table | pulp | 99 MB |

public | pulp_app_repository | table | pulp | 16 kB |

public | pulp_app_repositorycontent | table | pulp | 53 MB |

public | pulp_app_repositoryversion | table | pulp | 8192 bytes |

public | pulp_app_reservedresource | table | pulp | 16 kB |

public | pulp_app_task | table | pulp | 16 kB |

public | pulp_app_taskreservedresource | table | pulp | 8192 bytes |

public | pulp_app_worker | table | pulp | 40 kB |

(43 rows)