Project

Profile

Help

Issue #9000

closed

On-demand downloading fills up /var/run, causing errors

Added by dalley over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 101
Quarter:

Description

I have plenty of space in my VM - enough to sync centos7, centos8 baseos, and another 3gb of repos (total around 13gb) easily. And probably more beyond that.

If I on-demand sync my centos7-repo-into-file-repo and then hit it with locust to exercise the on-demand downloading, requests quickly start failing due to lack of space on the device, with only about half of the files downloaded.

Strangely /var/lib/pulp/ is not the culprit

(pulp) [vagrant@pulp3-source-fedora34 devel]$ du --summarize --human-readable /var/lib/pulp
1.5G    /var/lib/pulp

This is unrelated to caching, which was turned off.


Files


Related issues

Related to Pulp - Issue #8295: Disc Usage during Repository SyncCLOSED - CURRENTRELEASElmjachkyActions
Copied to Pulp - Backport #9110: Backport #9000 "On-demand downloading fills up /var/run, causing errors" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by dalley over 3 years ago

  • Description updated (diff)
Actions #2

Updated by dalley over 3 years ago

Perhaps /tmp is mounted in memory?

[vagrant@pulp3-source-fedora34 ~]$ free
               total        used        free      shared  buff/cache   available
Mem:         8143748     1527492      185568     1700732     6430688     4627912
Swap:              0           0           0
[vagrant@pulp3-source-fedora34 ~]$

The confusing part about this though is that the memory usage comes from the OS caching files, I would have thought that it would have been freed up on demand as stuff needed it. But this is just a theory, it might not be the cause.

Actions #3

Updated by dalley over 3 years ago

/tmp is indeed mounted in memory on Fedora, so that's the problem. Why is it not taking precedence over the OS file cache.

Actions #4

Updated by dalley over 3 years ago

  • Subject changed from File leak in on-demand downloading to On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors
Actions #5

Updated by dalley over 3 years ago

tmpfs and page cache are apparently essentially the same thing underneath the hood. so they both are counted by htop and "free" as cache.

Actions #6

Updated by dalley over 3 years ago

  • Subject changed from On-demand downloading uses tmpfs on many distros, tmpfs (apparently) is lower priority than the OS filesystem cache, causing errors to On-demand downloading fills up /var/run, causing errors

It's /var/run that is the problem

(pulp) [vagrant@pulp3-source-fedora34 devel]$ df -Th
Filesystem          Type        Size  Used Avail Use% Mounted on
devtmpfs            devtmpfs    3.9G     0  3.9G   0% /dev
tmpfs               tmpfs       3.9G  100K  3.9G   1% /dev/shm
tmpfs               tmpfs       1.6G  1.6G     0 100% /run
/dev/vda1           ext4         40G  4.6G   33G  13% /
tmpfs               tmpfs       3.9G  4.0K  3.9G   1% /tmp
tmpfs               tmpfs       796M  8.0K  796M   1% /run/user/1000
:/home/dalley/devel fuse.sshfs  858G  659G  156G  81% /home/vagrant/devel

/var/run/ is only 1.6gb (on my system!), we're filling it up very quickly with a bunch of files in /var/run/pulpcore-content/

So we need to evaluate our sizing guidance and/or our usage of this working directory (if it's not meant for this sort of activity, which I'm guessing is the case based on the low default size).

Actions #7

Updated by dalley over 3 years ago

  • Related to Issue #8295: Disc Usage during Repository Sync added
Actions #8

Updated by lmjachky over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky
Actions #9

Updated by dalley over 3 years ago

  • Sprint set to Sprint 100
Actions #10

Updated by dkliban@redhat.com over 3 years ago

  • Triaged changed from No to Yes
Actions #11

Updated by lmjachky over 3 years ago

Side notes:

  1. Running prestart clears the /var/run/pulpcore-content directory (obviously).
  2. Files present in the /var/run/pulpcore-content are temporary files that are used for saving artifacts straight away after serving content that was synced with the on_demand policy.
  3. The reported behaviour is caused by the fact that the whole repository's content had to be downloaded on-the-fly (due to the load testing) and the temporary files were not being removed.
  4. Similarly to the issue https://pulp.plan.io/issues/8295, the problem is in the way how we save artifacts and do not care about the removal of the temporary files.
  5. Extending the size of the /var/ partition is not a good solution and we should focus on Pulp and its handling of worthless temporary files.
Actions #12

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #13

Updated by pulpbot over 3 years ago

  • Status changed from ASSIGNED to POST
Actions #14

Updated by dalley over 3 years ago

  • Sprint/Milestone set to 3.15.0
Actions #15

Updated by dalley over 3 years ago

  • Copied to Backport #9110: Backport #9000 "On-demand downloading fills up /var/run, causing errors" to 3.14.z added

Added by Lubos Mjachky over 3 years ago

Revision f8bd7f8a | View on GitHub

Change content app's working directory dynamically

As of this commit, content app is no longer storing temporary files in the /var/run/ directory. The temporary files were created during on-demand downloading and were not removed until, e.g., restarting pulp services.

closes #9000

Actions #16

Updated by Anonymous over 3 years ago

  • Status changed from POST to MODIFIED
Actions #17

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF