RPM Support

Steps to reproduce the issues:

1. Preparing test data. Uploading 30,000 small ISOs to Pulp.
2. Upload an ISO to Pulp and capture elapsed time. (It takes around 16 seconds in my test env. After applying the fix, it just takes around 2 - 3 seconds.)

Analysis for the performance issue.

Let’s look at below piece of codes which is from “pulp/server/controllers/repository.py”.
def find_repo_content_units(..., repo_content_unit_q=None,..)
......
qs = model.RepositoryContentUnit.objects(q_obj=repo_content_unit_q,
repo_id=repository.repo_id)
......
for repo_content_unit in qs:
id_set = type_map.setdefault(repo_content_unit.unit_type_id, set())
id_set.add(repo_content_unit.unit_id)
content_unit_set = content_units.setdefault(repo_content_unit.unit_type_id, dict())
content_unit_set[repo_content_unit.unit_id] = repo_content_unit
......

As the value of parameter “repo_content_unit_q” equals “None” (invoker does not pass any data for it), "qs = model.RepositoryContentUnit.objects(q_obj=repo_content_unit_q, repo_id=repository.repo_id)" fetches all records of the repo from repo_content_units in MongoDB, then saves to Python a Map object and List objects.

Profiling shows the piece takes >80% of total time of uploading an unit (there are more than 30000 units in the repo). The major reason is result set “qs” is too large. Worse, “qs” will get larger as uploading more units, and performance will get worse and worse.

So passing a search criteria[1] "repo_content_unit_q" to filter out units as need can improve performance.

[1]https://github.com/pulp/pulp_rpm/pull/1236