Project

Profile

Help

Maintenance: Planio will be observing a scheduled maintenance window this Tuesday, November 5, 2024 from 03:00 UTC until 06:30 UTC to perform urgent network maintenance in our primary data center. Your Planio account will be unavailable during this maintenance window.

Issue #8295

closed

Disc Usage during Repository Sync

Added by wibbit over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 101
Quarter:
Q2-2021

Description

While performing a repository sync recently I noticed that the WORKING_DIRECTORY was of a much much larger size than I expected.

It is currently assigned 150GB, yet failed to sync 3 repositories (I believe RHEL7 Source, RHEL7 Debug, and possibly RHEL7 Server) due to running out of disk space.

I was of the understanding that the working directory was not intended to hold the entire repository till completion, but only the individual unit's being transferred, on a successful transfer, that would then be ingested and end up as an artefact and stored in the artefact's path.

I believe an additional side affect of this, is that though units have been downloaded cleanly, and I believe could have been converted to an artefact, this has not been done, and a complete re-download is required as evidenced by the fact that on an additional sync the same behaviour occurs.

Maybe this would be the case any way, unless the artefact is converted to content, I'm unsure of those specifics.


Files

sync_disk_usage_change.patch (2.15 KB) sync_disk_usage_change.patch lmjachky, 05/02/2021 01:54 PM

Related issues

Related to Pulp - Issue #8296: Pulp worker directories not cleaned upCLOSED - DUPLICATEActions
Related to Pulp - Issue #7316: Files are not being deleted from storage when calling the method delete()CLOSED - CURRENTRELEASEbmbouterActions
Related to Pulp - Issue #9000: On-demand downloading fills up /var/run, causing errorsCLOSED - CURRENTRELEASElmjachkyActions
Copied to Pulp - Backport #9103: Backport 8295 "Disc Usage during Repository Sync" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by dalley over 3 years ago

  • Project changed from RPM Support to Pulp
Actions #2

Updated by dalley over 3 years ago

  • Related to Issue #8296: Pulp worker directories not cleaned up added
Actions #3

Updated by dalley over 3 years ago

  • Related to Issue #7316: Files are not being deleted from storage when calling the method delete() added
Actions #4

Updated by daviddavis over 3 years ago

  • Triaged changed from No to Yes
Actions #5

Updated by ttereshc over 3 years ago

Just for some context. It might be a pulpcore issue but it also can be pulp_rpm specific because working directory is recreated a lot in pulp_rpm.

Actions #6

Updated by ipanova@redhat.com over 3 years ago

  • Sprint set to Sprint 94
Actions #7

Updated by lmjachky over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky
Actions #8

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 94 to Sprint 95
Actions #9

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #10

Updated by lmjachky over 3 years ago

I made a couple of experiments and noticed that our pipeline is not really creating artifacts from temporary files along the way during the sync process. This is caused by the parameter minsize (https://github.com/pulp/pulpcore/blob/354383883032277e7a1f7dc7ddf2dc0a5bc40fad/pulpcore/plugin/stages/api.py#L84) that retains the pipeline from saving artifacts unless the queue is filled up (https://github.com/pulp/pulpcore/blob/eda2c890214a76e6f6ffa18cc939d04273b1fa13/pulpcore/plugin/stages/artifact_stages.py#L229).

Furthermore, all the temporary files are being deleted only when the whole sync process ends. Because of the parameter delete in NamedTemporaryFile (https://github.com/pulp/pulpcore/blob/354383883032277e7a1f7dc7ddf2dc0a5bc40fad/pulpcore/download/base.py#L122), we should handle the removal manually by ourselves. The working directory (temporary files stored under a specific task) is cleared as a whole only when the sync process reasonably finishes.

Also, another problem is that in some cases pulp_rpm do not handle the removal of temporary repodata files at all. When I synced from the http://mirrors.sonic.net/epel/playground/8/Everything/x86_64/os/, the downloaded repodata (3.9GB) were not removed even when the sync process successfuly ended. It seemed to me that I never reached this return statement: https://github.com/pulp/pulp_rpm/blob/f14be05d58f06e794676a5222b443cd9d082f031/pulp_rpm/app/tasks/synchronizing.py#L428, weird. Yet, the sync proceeded without failures.

So, I propose to examine some of the default parameters we changed in https://github.com/pulp/pulpcore/pull/440/files and call os.unlink right after we create artifacts. I am attaching some of the advised changes in the description and unassigning myself.

Actions #11

Updated by lmjachky over 3 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (lmjachky)
Actions #12

Updated by rchan over 3 years ago

  • Sprint deleted (Sprint 96)
  • Quarter set to Q2-2021

We are putting this one on hold for now since this Sprint was busy, but do plan to work on it this quarter.

Actions #13

Updated by dalley over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky
  • Sprint set to Sprint 99

@Lubos, This is important work and we want to get to this soon (discussed in the 3 month planning meeting), but nobody has had time to pick it up yet. If you could pick it back up when you return next week that would be excellent

Actions #14

Updated by pulpbot over 3 years ago

  • Status changed from ASSIGNED to POST
Actions #15

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #16

Updated by lmjachky over 3 years ago

The commit https://github.com/pulp/pulpcore/commit/887dafa189639e310b7bd78ab96adbb188aa5a66 partially resolved the problem. Since it was merged, all temporary directories are properly cleared. Therefore, the statement "in some cases pulp_rpm do not handle the removal of temporary repodata files at all" is not true anymore.

However, the disk space is freed only when a task finishes. So, temporary files are still not being removed gradually through the sync, while the task is running. I am investigating more possibilities of resolving this issue.

Actions #18

Updated by dalley over 3 years ago

  • Related to Issue #9000: On-demand downloading fills up /var/run, causing errors added
Actions #19

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #21

Updated by dalley over 3 years ago

  • Sprint/Milestone set to 3.15.0
Actions #22

Updated by dalley over 3 years ago

  • Copied to Backport #9103: Backport 8295 "Disc Usage during Repository Sync" to 3.14.z added

Added by Lubos Mjachky over 3 years ago

Revision fbe2d7f9 | View on GitHub

Move downloaded files instead of copying them

Before this commit, the condition that checks whether pulp's temporary files are stored within /var/lib/pulp was never met because it looked for files starting with /var/lib/pulp/media (MEDIA_ROOT). MEDIA_ROOT is however no longer pointing to /var/lib/pulp.

closes #8295

Actions #23

Updated by Anonymous over 3 years ago

  • Status changed from POST to MODIFIED
Actions #24

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF