Project

Profile

Help

Issue #9000

On-demand downloading fills up /var/run, causing errors

Added by dalley 3 months ago. Updated 21 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 101
Quarter:

Description

I have plenty of space in my VM - enough to sync centos7, centos8 baseos, and another 3gb of repos (total around 13gb) easily. And probably more beyond that.

If I on-demand sync my centos7-repo-into-file-repo and then hit it with locust to exercise the on-demand downloading, requests quickly start failing due to lack of space on the device, with only about half of the files downloaded.

Strangely /var/lib/pulp/ is not the culprit

(pulp) [vagrant@pulp3-source-fedora34 devel]$ du --summarize --human-readable /var/lib/pulp
1.5G    /var/lib/pulp

This is unrelated to caching, which was turned off.


Related issues

Related to Pulp - Issue #8295: Disc Usage during Repository SyncCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to Pulp - Backport #9110: Backport #9000 "On-demand downloading fills up /var/run, causing errors" to 3.14.zCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision f8bd7f8a View on GitHub
Added by Lubos Mjachky about 2 months ago

Change content app's working directory dynamically

As of this commit, content app is no longer storing temporary files in the /var/run/ directory. The temporary files were created during on-demand downloading and were not removed until, e.g., restarting pulp services.

closes #9000

History

#1 Updated by dalley 3 months ago

  • Description updated (diff)

#2 Updated by dalley 3 months ago

Perhaps /tmp is mounted in memory?

[vagrant@pulp3-source-fedora34 ~]$ free
               total        used        free      shared  buff/cache   available
Mem:         8143748     1527492      185568     1700732     6430688     4627912
Swap:              0           0           0
[vagrant@pulp3-source-fedora34 ~]$

The confusing part about this though is that the memory usage comes from the OS caching files, I would have thought that it would have been freed up on demand as stuff needed it. But this is just a theory, it might not be the cause.

#3 Updated by dalley 3 months ago

/tmp is indeed mounted in memory on Fedora, so that's the problem. Why is it not taking precedence over the OS file cache.

#4 Updated by dalley 3 months ago

  • Subject changed from File leak in on-demand downloading to On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors

#5 Updated by dalley 3 months ago

tmpfs and page cache are apparently essentially the same thing underneath the hood. so they both are counted by htop and "free" as cache.

#6 Updated by dalley 3 months ago

  • Subject changed from On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors to On-demand downloading fills up /var/run, causing errors

It's /var/run that is the problem

(pulp) [vagrant@pulp3-source-fedora34 devel]$ df -Th
Filesystem          Type        Size  Used Avail Use% Mounted on
devtmpfs            devtmpfs    3.9G     0  3.9G   0% /dev
tmpfs               tmpfs       3.9G  100K  3.9G   1% /dev/shm
tmpfs               tmpfs       1.6G  1.6G     0 100% /run
/dev/vda1           ext4         40G  4.6G   33G  13% /
tmpfs               tmpfs       3.9G  4.0K  3.9G   1% /tmp
tmpfs               tmpfs       796M  8.0K  796M   1% /run/user/1000
:/home/dalley/devel fuse.sshfs  858G  659G  156G  81% /home/vagrant/devel

/var/run/ is only 1.6gb (on my system!), we're filling it up very quickly with a bunch of files in /var/run/pulpcore-content/

So we need to evaluate our sizing guidance and/or our usage of this working directory (if it's not meant for this sort of activity, which I'm guessing is the case based on the low default size).

#7 Updated by dalley 2 months ago

  • Related to Issue #8295: Disc Usage during Repository Sync added

#8 Updated by lmjachky 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky

#9 Updated by dalley 2 months ago

  • Sprint set to Sprint 100

#10 Updated by dkliban@redhat.com 2 months ago

  • Triaged changed from No to Yes

#11 Updated by lmjachky 2 months ago

Side notes:

  1. Running prestart clears the /var/run/pulpcore-content directory (obviously).
  2. Files present in the /var/run/pulpcore-content are temporary files that are used for saving artifacts straight away after serving content that was synced with the on_demand policy.
  3. The reported behaviour is caused by the fact that the whole repository's content had to be downloaded on-the-fly (due to the load testing) and the temporary files were not being removed.
  4. Similarly to the issue https://pulp.plan.io/issues/8295, the problem is in the way how we save artifacts and do not care about the removal of the temporary files.
  5. Extending the size of the /var/ partition is not a good solution and we should focus on Pulp and its handling of worthless temporary files.

#12 Updated by rchan 2 months ago

  • Sprint changed from Sprint 100 to Sprint 101

#13 Updated by pulpbot 2 months ago

  • Status changed from ASSIGNED to POST

#14 Updated by dalley about 2 months ago

  • Sprint/Milestone set to 3.15.0

#15 Updated by dalley about 2 months ago

  • Copied to Backport #9110: Backport #9000 "On-demand downloading fills up /var/run, causing errors" to 3.14.z added

#16 Updated by Anonymous about 2 months ago

  • Status changed from POST to MODIFIED

#17 Updated by pulpbot 21 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF