RPM Support

+1 to a cprofile collection as a next step. It would be great to see where all that time is going.

Looking at the cprofile results, it seems that a bulk of time is going towards making calls to the database (200 seconds) but per call, it's only about 0.00-0.02 seconds. I am going to take a look at reducing the number of sql calls we're making. I'm hoping to prefetch some of the data.

As for sqlite generation, removing that code saved about 46 seconds (441.148s vs 395.415s). I don't see a huge performance benefit in removing it although I am not opposed to doing so.

Using prefetching and bulk create I was able to speed up the publish task code from about 440 to 120 seconds.

I left the sqlite generation code in place but again, I will be ok to remove it.

I double checked with Katello and publishing epel7 (13,190 packages) in <= 2 min would meet their requirements.

I monitored the memory while running a publish on epel7. Here's a representative snapshot I saw:

PID/USER/PR/NI/VIRT/RES/SHR/S/%CPU/%MEM/TIME/COMMAND
29923 vagrant   20   0 1061332 888876   7872 R  82.0  22.0   1:08.23 rq         

Looks like it consumed about 900 MB (was 500 MB before). We could add batching later if it memory becomes a problem.