File Support

Content can not be used with bulk_create because it uses multi-table inheritance. An exception is raised when this is tried.

RemoteArtifact and ContentArtifact both can be used with bulk_create. My code does that successfully.

I encourage you to put performance requirements into context. This experiment revealed that when writing data to postgresql vs mongodb, there is a real cost to using a database with ACID guarantees and to writing more individual records per content unit. (I don't know if one or both of those is responsible, but generally doing more work with more features takes more time.) But: are you optimizing Pulp 3 for the ability to sync 70k 0-byte files from localhost?

If we started making those files larger, as the amount of time it takes to download the files approaches the amount of time it takes to write records to the database, the user's wait time attributable to the database approaches zero. An example will help.

At 70k units in 44 minutes, that's 26.5 units per second. What if each unit were 1MB in size on average? As long as downloading happens no faster than 26.5MBps (212Mbps), database writes would no longer be the limiting factor. Of course this assumes that database writes and downloading are happening concurrently, but I think Pulp has that problem mostly solved. In other words, if you made these files 1MB on average and put them on a remote machine across the internet, I bet the performance difference between Pulp 2 and 3 would look very different. Pulp 3 might even win.

Of course there are more variables. There could be unrelated database activity slowing our database writes, or unrelated network activity slowing our downloads. Some files might already exist and wouldn't need to be downloaded again.

So the question is: how much faster does your database write performance need to be than your download performance in order to be comfortable that most (you pick a percentage) users won't be waiting on their database? Pick your margin of comfort. What if, in a "normal case", the database is 1.5x faster than the fastest expected download speed? 2x? 3x?

You may need to choose:

• max download speed you are willing to optimize for. "If your connection is faster than X, wow, lucky you! You can wait a few minutes on the database."
• max portion of already-downloaded content you will optimize for. Again, if you happen to have more than X% of files already present on disk, what good fortune! This time you'll need to wait on the DB.
• expected average size of a content unit's files
• how much margin you want to build in for worse-than-normal database performance. 1.5x? 2x? 3x? 5x?

The tests in above comments are fine scenarios to use for improving your database write performance. That's a great thing to do, and it sounds like it revealed some opportunities. But before you set any hard performance requirements for Pulp 3 GA, I suggest picking those benchmarks carefully.

I also thought that any increases in runtime due to database operations being slower in Pulp3 would be offset by the huge amount of time that would be spent downloading. Then @jsherrill on irc pointed out to me that lazy downloading is the default for Katello, which is why they made the test with 0-byte files. This test is effectively the common case, which is my motivation to make it a Pulp3 requirement. The good news is I believe this goal is very achievable.

Here is a revised goal statement specifically highlighting that it's about lazy sync:

Pulp3 lazy sync of pulp_file should be roughly as fast on Pulp3 as Pulp2.

I believe this would make sense as a P1 item for the gap fit on https://projects.theforeman.org/projects/katello/wiki/PulpV3GapAnalysis#Core-Problem-Statements

Yes, i agree that we may end up seeing some sort of performance degregation with inserts using an ACID database, and filed this for some investigation. Initial investigation by brian indicates that this was not due solely to inherent slowness with the database, but instead with design decisions that had been made around how to use the database and django.

It seems important to address these structural changes before release if ever at all. In addition, what Brian said is correct, the default configuration of Katello and Satellite 6 is to use on_demand for yum repositories, so the zero byte test was somewhat of a good test for that scenario.

For this reason I think it is a P1 issue, until we either address the structural changes or determine that addressing those sub-optimial design decisions is either not possible or not worthwhile.

+1 to making this a P1.

I added this to the Gap Areas as a P1 here: https://projects.theforeman.org/projects/katello/wiki/PulpV3GapAnalysis#General-Issues

On my PC (with a fast SSD), I get an 18 minute sync for Pulp 2, and also an 18 minute sync for Pulp 3 with a minor modification to increase the batch size. I plan to add that modification to core this afternoon.

So it looks like we have performance parity for the file plugin, for "fake" lazy sync. Once the real lazy sync work is complete we can repeat this test a few times to see where we stand.

I will also test the single-table content branch but it's a good sign that even master branch is getting decent performance without having had much in the way of focused optimization yet. I may also try a few other low-hanging fruit optimizations on master branch to get a clearer picture when comparing against the single-table content branch, since that represents a fairly significant change.

I think this has caused so many good changes in our API and python API that as long as the performance is on-par I'm +1 on accepting it.

The performance gap has been seemingly rectified as of the latest master branch.

Last night I did a lazy sync of EPEL with Pulp 2 and Pulp 3, with the result that Pulp 3 only took 40% as long to perform the lazy sync (2 minutes 15 seconds vs 5 minutes 16 seconds). Actual sync performance as mentioned also seems to be comparable when using many very small files (specifically, 70k files each containing only a UUID).

One thing I'd like to test is downloading performance, so I may try to construct a file benchmark with larger files to make sure there's no regression there. If not, I think we should close this issue as-is and treat the single-table content work as a separate issue, presuming we still go forwards with it.

Here's the results of my performance testing

Tests with File sync, 70k small files served locally
----------------------------------------------------

Pulp 2 (master):     ~18 minutes (I didn't record the exact number)
Pulp 3 (master):     17m 47s

Pulp 3 (lazy sync):  3m 46s

Tests with RPM lazy sync against EPEL (https://dl.fedoraproject.org/pub/epel/7/x86_64/)
---------------------------------------------------------------------------------------

# Note: these numbers are from my laptop and not directly comparable to the ones above

Pulp 2 (master):  5m 16s
Pulp 3 (master):  2m 16s

Tests with RPM immediate sync against EPEL (https://dl.fedoraproject.org/pub/epel/7/x86_64/)
--------------------------------------------------------------------------------------------

Pulp 2 (master):  24m 56s
Pulp 3 (master):  21m 52s

Tests with RPM immediate sync against RPMFusion (https://download1.rpmfusion.org/free/fedora/updates/28/x86_64/)
----------------------------------------------------------------------------------------------------------------

Pulp 2 (master):  1m 03s
Pulp 3 (master):  0m 52s

Since Pulp 3 is now faster than Pulp 2 across the board (even before in-depth performance tuning), I'm going to close out this issue.