Story #4498
closed
As a user, I can use chunked uploading with S3
Status:
CLOSED - CURRENTRELEASE
Description
Currently, it is not possible to upload files to the S3 storage directly because files cannot be opened in the append mode. We use the local filesystem storage for dealing with chunked uploads (https://pulp.plan.io/issues/6253) at the moment. In terms of clustered deployment, pulp instances do not share a common filesystem.
We propose to use PulpTemporaryFile for storing each uploaded chunk separately in the S3 storage and then to assemble Artifact from the created instances of PulpTemporaryFile. In addition to that, we will allow users to select their own storage for uploading chunks.
This is required by https://pulp.plan.io/issues/7218.
- Related to Story #4196: As a user, I can upload files in chunks. added
- Sprint/Milestone deleted (
3.0.0)
- Groomed changed from No to Yes
- Description updated (diff)
- Status changed from NEW to ASSIGNED
- Assignee set to lmjachky
- Status changed from ASSIGNED to POST
At the moment, it is not possible to dynamically change the storage used for uploaded chunks without creating additional site-local migrations. Due to this, each uploaded chunk will be stored in the default file storage.
Support for callable storage was added in Django 3.1 and Pulp uses the Django 2.2 LTS version. Refer to https://docs.djangoproject.com/en/3.1/topics/files/#using-a-callable. If we decide to move to the next LTS version, we will be able to allow users to define their custom storage in the settings file.
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Sprint/Milestone set to 3.8.0
- Sprint/Milestone deleted (
3.8.0)
- Status changed from MODIFIED to ASSIGNED
We had to revert the commit due to the breaking changes for plugins.
The PR needs to be open again with the changes which won't break plugins.
settings.CHUNKED_UPLOAD_DIR change is problematic and needs to be done in stages.
- Status changed from ASSIGNED to POST
- Status changed from POST to MODIFIED
- Sprint/Milestone set to 3.9.0
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Also available in: Atom
PDF
.
Upload chunks as separate files
closes #4498