Re-migrations take nearly as long as initial migrations
With a large repository, re-migrations take a very long time.
<jsherrill> i was also planning on testing re-migration time with that repo <jsherrill> and that seemed really really slow <jsherrill> but i wasn't sure if that was related to the fact that not all of the units migrated properly <jsherrill> so was waiting for it to be fixed to test for sure <jsherrill> like it seemed like it took ~an hour to re-migrate it <jsherrill> i'm sure there is a lot of low-hanging fruit performance wise <jsherrill> and i'm less concerned about initial migration time
#2 Updated by dalley almost 2 years ago
Here's a flamegraph sampled from a few minutes of one of my migration runs, if it helps. Open it in firefox.
pip install py-spy sudo env "PATH=$PATH" py-spy record --pid 20183 --output ../migration_perf2.svg
I'm going to guess that we're saving the progress bars too much, just as we did with the other plugins. We're calling
pb.increment() in a loop for each content unit in a couple of places, which hugely inflates the # of DB queries.
#10 Updated by dalley over 1 year ago
- Status changed from POST to MODIFIED
Applied in changeset pulp:pulp-2to3-migration|c430b4c865753fbddb102334006a7adff8ae2f43.