As a user, I can use chunked uploading with S3
Currently, it is not possible to upload files to the S3 storage directly because files cannot be opened in the append mode. We use the local filesystem storage for dealing with chunked uploads (https://pulp.plan.io/issues/6253) at the moment. In terms of clustered deployment, pulp instances do not share a common filesystem.
We propose to use PulpTemporaryFile for storing each uploaded chunk separately in the S3 storage and then to assemble Artifact from the created instances of PulpTemporaryFile. In addition to that, we will allow users to select their own storage for uploading chunks.
This is required by https://pulp.plan.io/issues/7218.
Updated by lmjachky about 3 years ago
At the moment, it is not possible to dynamically change the storage used for uploaded chunks without creating additional site-local migrations. Due to this, each uploaded chunk will be stored in the default file storage.
Support for callable storage was added in Django 3.1 and Pulp uses the Django 2.2 LTS version. Refer to https://docs.djangoproject.com/en/3.1/topics/files/#using-a-callable. If we decide to move to the next LTS version, we will be able to allow users to define their custom storage in the settings file.
Updated by ttereshc about 3 years ago
- Status changed from MODIFIED to ASSIGNED
We had to revert the commit due to the breaking changes for plugins. The PR needs to be open again with the changes which won't break plugins.
settings.CHUNKED_UPLOAD_DIR change is problematic and needs to be done in stages.