Issue #9000
closed
On-demand downloading fills up /var/run, causing errors
Status:
CLOSED - CURRENTRELEASE
Description
I have plenty of space in my VM - enough to sync centos7, centos8 baseos, and another 3gb of repos (total around 13gb) easily. And probably more beyond that.
If I on-demand sync my centos7-repo-into-file-repo and then hit it with locust to exercise the on-demand downloading, requests quickly start failing due to lack of space on the device, with only about half of the files downloaded.
Strangely /var/lib/pulp/ is not the culprit
(pulp) [vagrant@pulp3-source-fedora34 devel]$ du --summarize --human-readable /var/lib/pulp
1.5G /var/lib/pulp
This is unrelated to caching, which was turned off.
Files
- Description updated (diff)
Perhaps /tmp is mounted in memory?
[vagrant@pulp3-source-fedora34 ~]$ free
total used free shared buff/cache available
Mem: 8143748 1527492 185568 1700732 6430688 4627912
Swap: 0 0 0
[vagrant@pulp3-source-fedora34 ~]$
The confusing part about this though is that the memory usage comes from the OS caching files, I would have thought that it would have been freed up on demand as stuff needed it. But this is just a theory, it might not be the cause.
/tmp is indeed mounted in memory on Fedora, so that's the problem. Why is it not taking precedence over the OS file cache.
- Subject changed from File leak in on-demand downloading to On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors
tmpfs and page cache are apparently essentially the same thing underneath the hood. so they both are counted by htop and "free" as cache.
- Subject changed from On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors to On-demand downloading fills up /var/run, causing errors
It's /var/run that is the problem
(pulp) [vagrant@pulp3-source-fedora34 devel]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 3.9G 100K 3.9G 1% /dev/shm
tmpfs tmpfs 1.6G 1.6G 0 100% /run
/dev/vda1 ext4 40G 4.6G 33G 13% /
tmpfs tmpfs 3.9G 4.0K 3.9G 1% /tmp
tmpfs tmpfs 796M 8.0K 796M 1% /run/user/1000
:/home/dalley/devel fuse.sshfs 858G 659G 156G 81% /home/vagrant/devel
/var/run/ is only 1.6gb (on my system!), we're filling it up very quickly with a bunch of files in /var/run/pulpcore-content/
So we need to evaluate our sizing guidance and/or our usage of this working directory (if it's not meant for this sort of activity, which I'm guessing is the case based on the low default size).
- Related to Issue #8295: Disc Usage during Repository Sync added
- Status changed from NEW to ASSIGNED
- Assignee set to lmjachky
- Triaged changed from No to Yes
Side notes:
- Running
prestart clears the /var/run/pulpcore-content directory (obviously).
- Files present in the
/var/run/pulpcore-content are temporary files that are used for saving artifacts straight away after serving content that was synced with the on_demand policy.
- The reported behaviour is caused by the fact that the whole repository's content had to be downloaded on-the-fly (due to the load testing) and the temporary files were not being removed.
- Similarly to the issue https://pulp.plan.io/issues/8295, the problem is in the way how we save artifacts and do not care about the removal of the temporary files.
- Extending the size of the
/var/ partition is not a good solution and we should focus on Pulp and its handling of worthless temporary files.
- Sprint changed from Sprint 100 to Sprint 101
- Status changed from ASSIGNED to POST
- Sprint/Milestone set to 3.15.0
- Copied to Backport #9110: Backport #9000 "On-demand downloading fills up /var/run, causing errors" to 3.14.z added
- Status changed from POST to MODIFIED
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Also available in: Atom
PDF
.
Change content app's working directory dynamically
As of this commit, content app is no longer storing temporary files in the /var/run/ directory. The temporary files were created during on-demand downloading and were not removed until, e.g., restarting pulp services.
closes #9000