Project

Profile

Help

Story #4498

As a user, I can use chunked uploading with S3

Added by daviddavis over 1 year ago. Updated 18 days ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Sprint:
Sprint 81
Quarter:

Description

Currently, it is not possible to upload files to the S3 storage directly because files cannot be opened in the append mode. We use the local filesystem storage for dealing with chunked uploads (https://pulp.plan.io/issues/6253) at the moment. In terms of clustered deployment, pulp instances do not share a common filesystem.

We propose to use PulpTemporaryFile for storing each uploaded chunk separately in the S3 storage and then to assemble Artifact from the created instances of PulpTemporaryFile. In addition to that, we will allow users to select their own storage for uploading chunks.

This is required by https://pulp.plan.io/issues/7218.


Related issues

Related to Pulp - Story #4196: As a user, I can upload files in chunks.CLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision e46c3ff7 View on GitHub
Added by Lubos Mjachky about 1 month ago

Upload chunks as separate files

closes #4498

Revision 1b6c7360 View on GitHub
Added by Lubos Mjachky 18 days ago

Upload chunks as separate files

closes #4498

History

#1 Updated by daviddavis over 1 year ago

  • Related to Story #4196: As a user, I can upload files in chunks. added

#2 Updated by bmbouter over 1 year ago

  • Tags deleted (Pulp 3)

#3 Updated by daviddavis about 1 year ago

  • Sprint/Milestone deleted (3.0.0)

#4 Updated by lmjachky 3 months ago

  • Groomed changed from No to Yes

#5 Updated by lmjachky 3 months ago

  • Description updated (diff)

#6 Updated by lmjachky 3 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky

#7 Updated by lmjachky 3 months ago

  • Sprint set to Sprint 81

#8 Updated by pulpbot 2 months ago

  • Status changed from ASSIGNED to POST

#9 Updated by lmjachky about 2 months ago

At the moment, it is not possible to dynamically change the storage used for uploaded chunks without creating additional site-local migrations. Due to this, each uploaded chunk will be stored in the default file storage.

Support for callable storage was added in Django 3.1 and Pulp uses the Django 2.2 LTS version. Refer to https://docs.djangoproject.com/en/3.1/topics/files/#using-a-callable. If we decide to move to the next LTS version, we will be able to allow users to define their custom storage in the settings file.

#10 Updated by Anonymous about 1 month ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#11 Updated by ttereshc about 1 month ago

  • Sprint/Milestone set to 3.8.0

#12 Updated by ttereshc about 1 month ago

  • Sprint/Milestone deleted (3.8.0)

#13 Updated by ttereshc about 1 month ago

  • Status changed from MODIFIED to ASSIGNED

We had to revert the commit due to the breaking changes for plugins. The PR needs to be open again with the changes which won't break plugins.

settings.CHUNKED_UPLOAD_DIR change is problematic and needs to be done in stages.

#14 Updated by pulpbot about 1 month ago

  • Status changed from ASSIGNED to POST

#15 Updated by Anonymous 18 days ago

  • Status changed from POST to MODIFIED

Please register to edit this issue

Also available in: Atom PDF