Issue #6111
closed
Re-migrations take nearly as long as initial migrations
Description
With a large repository, re-migrations take a very long time.
<jsherrill> i was also planning on testing re-migration time with that repo
<jsherrill> and that seemed really really slow
<jsherrill> but i wasn't sure if that was related to the fact that not all of the units migrated properly
<jsherrill> so was waiting for it to be fixed to test for sure
<jsherrill> like it seemed like it took ~an hour to re-migrate it
<jsherrill> i'm sure there is a lot of low-hanging fruit performance wise
<jsherrill> and i'm less concerned about initial migration time
Files
Related issues
Updated by dalley almost 5 years ago
- File migration_perf2.svg migration_perf2.svg added
Here's a flamegraph sampled from a few minutes of one of my migration runs, if it helps. Open it in firefox.
Captured with:
pip install py-spy
sudo env "PATH=$PATH" py-spy record --pid 20183 --output ../migration_perf2.svg
I'm going to guess that we're saving the progress bars too much, just as we did with the other plugins. We're calling pb.increment() in a loop for each content unit in a couple of places, which hugely inflates the # of DB queries.
Updated by ipanova@redhat.com almost 5 years ago
- Related to Task #6156: Improve performance by cutting way down on ProgressReport updating added
Updated by ggainey over 4 years ago
- Tags Katello added
- Tags deleted (
Katello-P3)
Updated by dalley over 4 years ago
- Status changed from NEW to POST
- Assignee set to dalley
Added by dalley over 4 years ago
Added by dalley over 4 years ago
Revision c430b4c8 | View on GitHub
Don't iterate all lazy content every time we re-migrate
Push some of our Python-level checks into database query logic to significantly reduce the # of individual queries we make and the amount of content we need to iterate over / prefetch for.
Added by dalley over 4 years ago
Revision c430b4c8 | View on GitHub
Don't iterate all lazy content every time we re-migrate
Push some of our Python-level checks into database query logic to significantly reduce the # of individual queries we make and the amount of content we need to iterate over / prefetch for.
Updated by dalley over 4 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp:pulp-2to3-migration|c430b4c865753fbddb102334006a7adff8ae2f43.
Updated by ttereshc about 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Don't iterate all lazy content every time we re-migrate
Push some of our Python-level checks into database query logic to significantly reduce the # of individual queries we make and the amount of content we need to iterate over / prefetch for.
closes: #6111 https://pulp.plan.io/issues/6111