Pulp

Pulp 3 should accept very large files (~10GB or more) via chunked uploads like Pulp 2 does, and the API documentation should detail how to do this.

I just tested the same command directly on the django web server and I was able to upload 14mb file.

This seems to be a problem with the nginx config provided by the installer.

I emailed the author of django-chunked-upload since it's been a year since code has been merged in the repository. Waiting to hear back.

Here's another dead uploader project: https://github.com/douglasmiranda/django-fine-uploader

I found another package that seems to be (more?) active. It fits our needs better since it integrates with DRF (as opposed to django). It's also based on django-chunked-upload. I'm going to try to use it.

https://github.com/jkeifer/drf-chunked-upload

Looking at the django-chunked-uploader package, the workflow to create an artifact from a chunked upload would be something like the following. Assume we have 3 file chunks.

$ http --form PUT https://pulp3/8000/pulp/api/v3/uploads/ file@./chunk1
# {"url": "https://pulp3:8000/pulp/api/v3/uploads/5230ec1f59d1485d9d7974b853802e31", "offset": 10000, "expires": "2019-03-18T17:56:22.186Z"}
$ http --form PUT https://pulp3:8000/pulp/api/v3/uploads/5230ec1f59d1485d9d7974b853802e31 file@./chunk2
$ http --form PUT https://pulp3:8000/pulp/api/v3/uploads/5230ec1f59d1485d9d7974b853802e31 file@./chunk3 
$ http POST https://pulp3:8000/pulp/api/v3/uploads/5230ec1f59d1485d9d7974b853802e31 md5=0d599f0ec05c3bda8c3b8a68c32a1b47
# POSTing md5 creates the file
$ http POST https://pulp3:8000/pulp/api/v3/artifacts/ upload_id=/pulp/api/v3/uploads/5230ec1f59d1485d9d7974b853802e31

This looks good to me.

It would be nice if there was a solution that could support parallel chunk uploads.

The django-fine-uploader package does but it's the most out of date package (hasn't been updated since April 2017) and has some gaps like Python 3 support. And also, it's not made to work with DRF so getting it to work with DRF will probably require some work.

Looks good to me too! I guess you don't have to worry about offsets and what not, you just upload chunks in a serial manner?

That's correct. Does Katello require parallel chunk uploads?

I'm thinking of using drf-chunked-upload and (eventually) opening a PR against it with support for parallel chunk uploads.

No, i would not say its in our requirements to upload multiple chunks of the same file at once.

@daviddavis your plan sounds good to me. There are some users who do value parallel chunk support for very large files, e.g. isos, but that can definitely come later. Thank you for looking into this upload feature.