Issue #9000
closed
On-demand downloading fills up /var/run, causing errors
Description
I have plenty of space in my VM - enough to sync centos7, centos8 baseos, and another 3gb of repos (total around 13gb) easily. And probably more beyond that.
If I on-demand sync my centos7-repo-into-file-repo and then hit it with locust to exercise the on-demand downloading, requests quickly start failing due to lack of space on the device, with only about half of the files downloaded.
Strangely /var/lib/pulp/ is not the culprit
(pulp) [vagrant@pulp3-source-fedora34 devel]$ du --summarize --human-readable /var/lib/pulp
1.5G /var/lib/pulp
This is unrelated to caching, which was turned off.
Files
| Screenshot from 2021-06-30 21-00-26.png (585 KB) Screenshot from 2021-06-30 21-00-26.png | |||
| Screenshot from 2021-06-30 21-10-32.png (457 KB) Screenshot from 2021-06-30 21-10-32.png |
Related issues
Updated by dalley over 3 years ago
Perhaps /tmp is mounted in memory?
[vagrant@pulp3-source-fedora34 ~]$ free
total used free shared buff/cache available
Mem: 8143748 1527492 185568 1700732 6430688 4627912
Swap: 0 0 0
[vagrant@pulp3-source-fedora34 ~]$
The confusing part about this though is that the memory usage comes from the OS caching files, I would have thought that it would have been freed up on demand as stuff needed it. But this is just a theory, it might not be the cause.
Updated by dalley over 3 years ago
/tmp is indeed mounted in memory on Fedora, so that's the problem. Why is it not taking precedence over the OS file cache.
Updated by dalley over 3 years ago
- Subject changed from File leak in on-demand downloading to On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors
Updated by dalley over 3 years ago
tmpfs and page cache are apparently essentially the same thing underneath the hood. so they both are counted by htop and "free" as cache.
Updated by dalley over 3 years ago
- Subject changed from On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors to On-demand downloading fills up /var/run, causing errors
It's /var/run that is the problem
(pulp) [vagrant@pulp3-source-fedora34 devel]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 3.9G 100K 3.9G 1% /dev/shm
tmpfs tmpfs 1.6G 1.6G 0 100% /run
/dev/vda1 ext4 40G 4.6G 33G 13% /
tmpfs tmpfs 3.9G 4.0K 3.9G 1% /tmp
tmpfs tmpfs 796M 8.0K 796M 1% /run/user/1000
:/home/dalley/devel fuse.sshfs 858G 659G 156G 81% /home/vagrant/devel
/var/run/ is only 1.6gb (on my system!), we're filling it up very quickly with a bunch of files in /var/run/pulpcore-content/
So we need to evaluate our sizing guidance and/or our usage of this working directory (if it's not meant for this sort of activity, which I'm guessing is the case based on the low default size).
Updated by dalley over 3 years ago
- Related to Issue #8295: Disc Usage during Repository Sync added
Updated by lmjachky over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to lmjachky
Updated by lmjachky over 3 years ago
Side notes:
- Running
prestartclears the/var/run/pulpcore-contentdirectory (obviously). - Files present in the
/var/run/pulpcore-contentare temporary files that are used for saving artifacts straight away after serving content that was synced with theon_demandpolicy. - The reported behaviour is caused by the fact that the whole repository's content had to be downloaded on-the-fly (due to the load testing) and the temporary files were not being removed.
- Similarly to the issue https://pulp.plan.io/issues/8295, the problem is in the way how we save artifacts and do not care about the removal of the temporary files.
- Extending the size of the
/var/partition is not a good solution and we should focus on Pulp and its handling of worthless temporary files.
Updated by pulpbot over 3 years ago
- Status changed from ASSIGNED to POST
Updated by dalley over 3 years ago
- Copied to Backport #9110: Backport #9000 "On-demand downloading fills up /var/run, causing errors" to 3.14.z added
Added by Lubos Mjachky over 3 years ago
Updated by Anonymous over 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulpcore|f8bd7f8a2e696d1f429b3aae2886afbcd66bc504.
Updated by pulpbot over 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Change content app's working directory dynamically
As of this commit, content app is no longer storing temporary files in the /var/run/ directory. The temporary files were created during on-demand downloading and were not removed until, e.g., restarting pulp services.
closes #9000