Issue #8295
Disc Usage during Repository Sync
Description
While performing a repository sync recently I noticed that the WORKING_DIRECTORY was of a much much larger size than I expected.
It is currently assigned 150GB, yet failed to sync 3 repositories (I believe RHEL7 Source, RHEL7 Debug, and possibly RHEL7 Server) due to running out of disk space.
I was of the understanding that the working directory was not intended to hold the entire repository till completion, but only the individual unit's being transferred, on a successful transfer, that would then be ingested and end up as an artefact and stored in the artefact's path.
I believe an additional side affect of this, is that though units have been downloaded cleanly, and I believe could have been converted to an artefact, this has not been done, and a complete re-download is required as evidenced by the fact that on an additional sync the same behaviour occurs.
Maybe this would be the case any way, unless the artefact is converted to content, I'm unsure of those specifics.
Related issues
History
#2
Updated by dalley 2 months ago
- Related to Issue #8296: Pulp worker directories not cleaned up added
#3
Updated by dalley 2 months ago
- Related to Issue #7316: Files are not being deleted from storage when calling the method delete() added
#4
Updated by daviddavis 2 months ago
- Triaged changed from No to Yes
#6
Updated by ipanova@redhat.com 29 days ago
- Sprint set to Sprint 94
#10
Updated by lmjachky 5 days ago
I made a couple of experiments and noticed that our pipeline is not really creating artifacts from temporary files along the way during the sync process. This is caused by the parameter minsize (https://github.com/pulp/pulpcore/blob/354383883032277e7a1f7dc7ddf2dc0a5bc40fad/pulpcore/plugin/stages/api.py#L84) that retains the pipeline from saving artifacts unless the queue is filled up (https://github.com/pulp/pulpcore/blob/eda2c890214a76e6f6ffa18cc939d04273b1fa13/pulpcore/plugin/stages/artifact_stages.py#L229).
Furthermore, all the temporary files are being deleted only when the whole sync process ends. Because of the parameter delete in NamedTemporaryFile (https://github.com/pulp/pulpcore/blob/354383883032277e7a1f7dc7ddf2dc0a5bc40fad/pulpcore/download/base.py#L122), we should handle the removal manually by ourselves. The working directory (temporary files stored under a specific task) is cleared as a whole only when the sync process reasonably finishes.
Also, another problem is that in some cases pulp_rpm do not handle the removal of temporary repodata files at all. When I synced from the http://mirrors.sonic.net/epel/playground/8/Everything/x86_64/os/, the downloaded repodata (3.9GB) were not removed even when the sync process successfuly ended. It seemed to me that I never reached this return statement: https://github.com/pulp/pulp_rpm/blob/f14be05d58f06e794676a5222b443cd9d082f031/pulp_rpm/app/tasks/synchronizing.py#L428, weird. Yet, the sync proceeded without failures.
So, I propose to examine some of the default parameters we changed in https://github.com/pulp/pulpcore/pull/440/files and call os.unlink right after we create artifacts. I am attaching some of the advised changes in the description and unassigning myself.
Please register to edit this issue
.