Story #4498
closed
As a user, I can use chunked uploading with S3
100%
Description
Currently, it is not possible to upload files to the S3 storage directly because files cannot be opened in the append mode. We use the local filesystem storage for dealing with chunked uploads (https://pulp.plan.io/issues/6253) at the moment. In terms of clustered deployment, pulp instances do not share a common filesystem.
We propose to use PulpTemporaryFile for storing each uploaded chunk separately in the S3 storage and then to assemble Artifact from the created instances of PulpTemporaryFile. In addition to that, we will allow users to select their own storage for uploading chunks.
This is required by https://pulp.plan.io/issues/7218.
Related issues
Updated by daviddavis almost 6 years ago
- Related to Story #4196: As a user, I can upload files in chunks. added
Updated by lmjachky over 4 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to lmjachky
Updated by pulpbot over 4 years ago
- Status changed from ASSIGNED to POST
Updated by lmjachky about 4 years ago
At the moment, it is not possible to dynamically change the storage used for uploaded chunks without creating additional site-local migrations. Due to this, each uploaded chunk will be stored in the default file storage.
Support for callable storage was added in Django 3.1 and Pulp uses the Django 2.2 LTS version. Refer to https://docs.djangoproject.com/en/3.1/topics/files/#using-a-callable. If we decide to move to the next LTS version, we will be able to allow users to define their custom storage in the settings file.
Added by Lubos Mjachky about 4 years ago
Updated by Anonymous about 4 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulpcore|e46c3ff7653a180cd6d778ad9bacf96194d3ef55.
Updated by ttereshc about 4 years ago
- Status changed from MODIFIED to ASSIGNED
We had to revert the commit due to the breaking changes for plugins. The PR needs to be open again with the changes which won't break plugins.
settings.CHUNKED_UPLOAD_DIR change is problematic and needs to be done in stages.
Updated by pulpbot about 4 years ago
- Status changed from ASSIGNED to POST
Added by Lubos Mjachky about 4 years ago
Revision 1b6c7360 | View on GitHub
Upload chunks as separate files
closes #4498
Updated by Anonymous about 4 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulpcore|1b6c7360c30cc7d9e2d3b9fc5062ac1f7d69c2de.
Updated by pulpbot about 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Upload chunks as separate files
closes #4498