Project

Profile

Help

Story #4498

closed

As a user, I can use chunked uploading with S3

Added by daviddavis almost 4 years ago. Updated almost 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Sprint:
Sprint 81
Quarter:

Description

Currently, it is not possible to upload files to the S3 storage directly because files cannot be opened in the append mode. We use the local filesystem storage for dealing with chunked uploads (https://pulp.plan.io/issues/6253) at the moment. In terms of clustered deployment, pulp instances do not share a common filesystem.

We propose to use PulpTemporaryFile for storing each uploaded chunk separately in the S3 storage and then to assemble Artifact from the created instances of PulpTemporaryFile. In addition to that, we will allow users to select their own storage for uploading chunks.

This is required by https://pulp.plan.io/issues/7218.


Related issues

Related to Pulp - Story #4196: As a user, I can upload files in chunks.CLOSED - CURRENTRELEASEdaviddavis

Actions
Actions #1

Updated by daviddavis almost 4 years ago

  • Related to Story #4196: As a user, I can upload files in chunks. added
Actions #2

Updated by bmbouter over 3 years ago

  • Tags deleted (Pulp 3)
Actions #3

Updated by daviddavis about 3 years ago

  • Sprint/Milestone deleted (3.0.0)
Actions #4

Updated by lmjachky over 2 years ago

  • Groomed changed from No to Yes
Actions #5

Updated by lmjachky over 2 years ago

  • Description updated (diff)
Actions #6

Updated by lmjachky about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky
Actions #7

Updated by lmjachky about 2 years ago

  • Sprint set to Sprint 81
Actions #8

Updated by pulpbot about 2 years ago

  • Status changed from ASSIGNED to POST
Actions #9

Updated by lmjachky about 2 years ago

At the moment, it is not possible to dynamically change the storage used for uploaded chunks without creating additional site-local migrations. Due to this, each uploaded chunk will be stored in the default file storage.

Support for callable storage was added in Django 3.1 and Pulp uses the Django 2.2 LTS version. Refer to https://docs.djangoproject.com/en/3.1/topics/files/#using-a-callable. If we decide to move to the next LTS version, we will be able to allow users to define their custom storage in the settings file.

Added by Lubos Mjachky about 2 years ago

Revision e46c3ff7

Upload chunks as separate files

closes #4498

Actions #10

Updated by Anonymous about 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #11

Updated by ttereshc about 2 years ago

  • Sprint/Milestone set to 3.8.0
Actions #12

Updated by ttereshc about 2 years ago

  • Sprint/Milestone deleted (3.8.0)
Actions #13

Updated by ttereshc about 2 years ago

  • Status changed from MODIFIED to ASSIGNED

We had to revert the commit due to the breaking changes for plugins. The PR needs to be open again with the changes which won't break plugins.

settings.CHUNKED_UPLOAD_DIR change is problematic and needs to be done in stages.

Actions #14

Updated by pulpbot about 2 years ago

  • Status changed from ASSIGNED to POST

Added by Lubos Mjachky about 2 years ago

Revision 1b6c7360

Upload chunks as separate files

closes #4498

Actions #15

Updated by Anonymous about 2 years ago

  • Status changed from POST to MODIFIED
Actions #16

Updated by daviddavis almost 2 years ago

  • Sprint/Milestone set to 3.9.0
Actions #17

Updated by pulpbot almost 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF