Issue #1702
closed
Fetching repo info with details takes much longer on 2.8
Status:
CLOSED - CURRENTRELEASE
Description
Hitting this api:
/pulp/api/v2/repositories/REPO_ID/?details=true
seems to take much longer on 2.8 than it did on 2.6.
On 2.8 I'm seeing it take 6 seconds on a repo with 20K units, 4 seconds on a repo with 11K units. On pulp 2.6 this seemed almost instant.
The result of this for katello is that some operations seem to take a much longer time than before. Some page views make this api call multiple times across different repos (once per repo).
I strongly suspect this is the repository controller's "missing_unit_count" function taking up the time, which determines how many units in the repo do not have their files downloaded.
- Priority changed from Normal to High
- Platform Release set to 2.8.0
- Triaged changed from No to Yes
- Priority changed from High to Normal
- Platform Release deleted (
2.8.0)
- Triaged changed from Yes to No
The functions used by missing_unit_count() use the default pagination size of 1000. In this case we're only querying by unit_id and downloaded=False which will produce a consistent and deterministic size query. Perhaps the pagination size for this query can be safely increased to improve performance? Rough calculations (not accounting for query overhead) suggest that the json representation of a query containing a list of 100K UUIDs has about a 4MB memory footprint. Even when considering query syntax overhead not accounted for - the data suggests we can safely increase the pagination size used when obtaining the count to something closer to 100K than 1K. I would hope that even 25K would be 25x faster. Should be easy to confirm with some experimentation.
- Status changed from NEW to ASSIGNED
- Assignee set to jortel@redhat.com
Tested with a repository containing 12k units. Two findings:
- get_associated_unit_ids() confirmed as responsible. The root cause is mongoengine creating Document objects for queried unit associations.
- changing page size to: 25k has no impact.
- Platform Release set to 2.8.0
- Status changed from ASSIGNED to POST
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Triaged changed from No to Yes
- Status changed from MODIFIED to 5
- Status changed from 5 to CLOSED - CURRENTRELEASE
We are removing the 'API' category per open floor discussion June 16, 2020.
Also available in: Atom
PDF
Fix performance regression in associated unit queries. The mongoengine object instantiation is too slow. closes #1702