Pulp

We should avoid cherry picking this issue to a z-stream branch and instead align it with the 3.7 release since it involves a major change in functionality for FileFields.

I believe we may be handling this already in our code (see HandleTempFilesMixin). It would be good to test this out and confirm though.

daviddavis wrote:

I believe we may be handling this already in our code (see HandleTempFilesMixin). It would be good to test this out and confirm though.

This was the main reason for introducing django-cleanup to our project: https://github.com/pulp/pulpcore/pull/844#discussion_r469379057. Still, I agree with the reassessment.

If the files are not being deleted for real, we will probably need to implement a custom signal handler, as I did here: https://github.com/pulp/pulp_container/pull/133/files.

I tested this and I can say that at least when you delete an Artifact it also deletes the data from the storage backend.

Here's a simple reproducer:

import os

import django
django.setup()

from pulpcore.plugin.models import Artifact

path = '/tmp/asdfsfsdfsdfs'
with open(path, 'w') as f:
    f.write('asdfasdfasdf')


artifact = Artifact.init_and_validate(path)
artifact.save()

storage_path = '/var/lib/pulp/media/artifact/42/1c76d77563afa1914846b010bd164f395bd34c2102e5e99e0cb9cf173c1d87'
if os.path.exists(storage_path):
    print(f"{storage_path} exists!")

artifact.delete()

if not os.path.exists(storage_path):
    print(f"{storage_path} was deleted!")

When I run that I get:

/var/lib/pulp/media/artifact/42/1c76d77563afa1914846b010bd164f395bd34c2102e5e99e0cb9cf173c1d87 exists!
/var/lib/pulp/media/artifact/42/1c76d77563afa1914846b010bd164f395bd34c2102e5e99e0cb9cf173c1d87 was deleted!

Additionally, if I sync a file repo against an empty Pulp system and tree the /var/lib/pulp/media/ folder I get:

tree /var/lib/pulp/media/
/var/lib/pulp/media/
└── artifact
    ├── 8a
    │   └── 46213a107d6dde3c5f0e74f2ded19a1a9767be2ec0d14d274f4908db8cb836
    ├── 98
    │   └── 420a2eb7a86766e2e8a0ec4461b4022f754359cd6a3f5f1192b2eaf7c62672
    ├── d4
    │   └── 1dd7b1699638bf81ee78f197f761ec18ec61d7bf566880f9c1c088e946d8b7
    └── de
        └── 1e4813fedff7ee0223d63c4aaba1b9b3399cbff0edf059083be7ab1a157f9d

5 directories, 4 files

If I then delete the publication and run orphan cleanup with:

pulp file publication destroy /pulp/api/v3/publications/file/file/f8bc0232-2dbd-472b-9f48-660fb648ad09/
pulp orphans delete

tree then shows me:

/var/lib/pulp/media/
└── artifact
    ├── 8a
    │   └── 46213a107d6dde3c5f0e74f2ded19a1a9767be2ec0d14d274f4908db8cb836
    ├── 98
    │   └── 420a2eb7a86766e2e8a0ec4461b4022f754359cd6a3f5f1192b2eaf7c62672
    ├── d4
    └── de
        └── 1e4813fedff7ee0223d63c4aaba1b9b3399cbff0edf059083be7ab1a157f9d

5 directories, 3 files

After that if I delete the repository and run orphan cleanup with:

pulp file repository destroy --name repo3692
pulp orphans delete

tree shows me all files are deleted:

/var/lib/pulp/media/
└── artifact
    ├── 8a
    ├── 98
    ├── d4
    └── de

5 directories, 0 files

Here is a test script to check whether deleting uploads are deleting upload chunk files:

curl -O https://fixtures.pulpproject.org/file-large/1.iso
split --bytes=6M 1.iso chunk
export UPLOAD=$(http POST :/pulp/api/v3/uploads/ size=`ls -l 1.iso | cut -d ' ' -f5` | jq -r '.pulp_href')
http --form PUT :$UPLOAD file@./chunkab 'Content-Range:bytes 6291456-10485759/*'
http --form PUT :$UPLOAD file@./chunkaa 'Content-Range:bytes 0-6291455/*'
http DELETE :$UPLOAD
ls /var/lib/pulp/media/upload

Using the reproduer from comment 26, I've confirmed that this is an issue. After the DELETE the /var/lib/pulp/media/upload/ dir does contain:

/var/lib/pulp/media/upload
├── 72df4f1f-9c92-45ad-9b9b-90f4e2a9b86a
└── 76ac4feb-9b61-4a24-88a3-e748c4563886

I'll try to resolve the deletion of those uploads also now.