On-demand downloading fills up /var/run, causing errors
I have plenty of space in my VM - enough to sync centos7, centos8 baseos, and another 3gb of repos (total around 13gb) easily. And probably more beyond that.
If I on-demand sync my centos7-repo-into-file-repo and then hit it with locust to exercise the on-demand downloading, requests quickly start failing due to lack of space on the device, with only about half of the files downloaded.
Strangely /var/lib/pulp/ is not the culprit
(pulp) [vagrant@pulp3-source-fedora34 devel]$ du --summarize --human-readable /var/lib/pulp 1.5G /var/lib/pulp
This is unrelated to caching, which was turned off.
Change content app's working directory dynamically
As of this commit, content app is no longer storing temporary files in the /var/run/ directory. The temporary files were created during on-demand downloading and were not removed until, e.g., restarting pulp services.
Perhaps /tmp is mounted in memory?
[vagrant@pulp3-source-fedora34 ~]$ free total used free shared buff/cache available Mem: 8143748 1527492 185568 1700732 6430688 4627912 Swap: 0 0 0 [vagrant@pulp3-source-fedora34 ~]$
The confusing part about this though is that the memory usage comes from the OS caching files, I would have thought that it would have been freed up on demand as stuff needed it. But this is just a theory, it might not be the cause.
- Subject changed from On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors to On-demand downloading fills up /var/run, causing errors
It's /var/run that is the problem
(pulp) [vagrant@pulp3-source-fedora34 devel]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 3.9G 0 3.9G 0% /dev tmpfs tmpfs 3.9G 100K 3.9G 1% /dev/shm tmpfs tmpfs 1.6G 1.6G 0 100% /run /dev/vda1 ext4 40G 4.6G 33G 13% / tmpfs tmpfs 3.9G 4.0K 3.9G 1% /tmp tmpfs tmpfs 796M 8.0K 796M 1% /run/user/1000 :/home/dalley/devel fuse.sshfs 858G 659G 156G 81% /home/vagrant/devel
/var/run/ is only 1.6gb (on my system!), we're filling it up very quickly with a bunch of files in /var/run/pulpcore-content/
So we need to evaluate our sizing guidance and/or our usage of this working directory (if it's not meant for this sort of activity, which I'm guessing is the case based on the low default size).
- Files present in the
/var/run/pulpcore-contentare temporary files that are used for saving artifacts straight away after serving content that was synced with the
- The reported behaviour is caused by the fact that the whole repository's content had to be downloaded on-the-fly (due to the load testing) and the temporary files were not being removed.
- Similarly to the issue https://pulp.plan.io/issues/8295, the problem is in the way how we save artifacts and do not care about the removal of the temporary files.
- Extending the size of the
/var/partition is not a good solution and we should focus on Pulp and its handling of worthless temporary files.
Please register to edit this issue