Project

Profile

Help

Test #5263

closed

Test - As a user, I can upload chunks in parallel

Added by kersom over 4 years ago. Updated about 2 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Version:
Platform Release:
Tags:
Sprint:
Quarter:

Description

Ticket moved to GitHub: "pulp/pulpcore/1842":https://github.com/pulp/pulpcore/issues/1842


We're currently using drf-chunked-uploads[0] but it seems like the library has become unmaintained[1] since we adopted. It has some other quirks and missing features too. So I think we should move off of it and roll our code as part of this story.

Solution

Add a design which supports sha256 and parallel uploads of chunks.

Models

Upload

id = UUID
file = File
size = BigIntegerField
user = FK
created_at = DateTimeField
completed_at = DateTimeField

UploadChunk

id = UUID
upload = FK
offset = BigIntegerField
size = BigIntegerField

Workflow

# create the upload session
http POST :24817/pulp/api/v3/uploads/ size=10485759 # returns a UUID (e.g. 345b7d58-f1f8-45d9-d354-82a31eb879bf)
export UPLOAD='/pulp/api/v3/uploads345b7d58-f1f8-45d9-d354-82a31eb879bf/'

# note the order doesn't matter here
http --form PUT :24817$UPLOAD file@./chunkab 'Content-Range:bytes 6291456-10485759/32095676'
http --form PUT :24817$UPLOAD file@./chunkaa 'Content-Range:bytes 0-6291455/32095676'

# view the upload and its chunks
http :24817${UPLOAD}

# complete the upload
http PUT :24817${UPLOAD}commit sha256=037a47d93670e64f2b1038e6f90e4cfd

# create the artifact from the upload
http POST :24817/pulp/api/v3/artifacts/ upload=$UPLOAD

Additional references

https://github.com/douglasmiranda/django-fine-uploader
https://medium.com/box-developer-blog/introducing-the-chunked-upload-api-f82c820ccfcb

[0] https://github.com/jkeifer/drf-chunked-upload
[1] https://github.com/jkeifer/drf-chunked-upload/pull/8


Related issues

Related to Pulp - Story #4488: As a user, I can upload chunks in parallelCLOSED - CURRENTRELEASEdaviddavis

Actions
Actions #1

Updated by kersom over 4 years ago

  • Related to Story #4488: As a user, I can upload chunks in parallel added
Actions #2

Updated by kersom over 4 years ago

  • Tags deleted (Katello-P1)
Actions #3

Updated by daviddavis over 4 years ago

  • Sprint/Milestone deleted (3.0.0)
Actions #4

Updated by pulpbot about 2 years ago

  • Description updated (diff)
  • Status changed from NEW to CLOSED - DUPLICATE

Also available in: Atom PDF