Project

Profile

Help

Story #2843

closed

As an authenticated user, I can create an Artifact by uploading a file

Added by dkliban@redhat.com over 6 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 23
Quarter:

Description

For an API user to create an Artifact, Pulp 3 needs to have the following:

- updated Artifact[0] model without a 'content' foreign key nor the 'relative_path' field. A uniqueness constraint should be added on the sha256 field.
- viewset that can handle CRD operations for Artifacts. It most likely needs to use the FileUploadParser[1] and a custom Django Upload Handler[2].
- serializer for the viewset which will return all serialized fields of the Artifact model. The file field should be it's full path on disk.
- API endpoint at /api/v3/content/artifacts
- POST request to the /api/v3/content/artifacts/ endpoint creates an Artifact. The body of the request contains multipart form data with the following:

file - The file being uploaded
size - The size of the file in bytes.
md5 - The MD5 checksum of the file.
sha1 - The SHA-1 checksum of the file.
sha224 - The SHA-224 checksum of the file.
sha256 - The SHA-256 checksum of the file.
sha384 - The SHA-384 checksum of the file.
sha512 - The SHA-512 checksum of the file.

Before the model is saved, a SHA-256 checksum (digest) of the uploaded file is generated.
If a sha256 was specified in POST parameters, the generated hash is validated against the value specified as the POST parameter. If the values don't match a validation exception is raised.
If an Artifact with the same sha256 checksum already exists, a 400 response is returned to the user.
When the model is saved, the file is written to MEDIA_ROOT/content/units/digest[0:2]/digest[2:]
After a successful save, a serialized version of the Artifact is returned.

- GET request to the /api/v3/content/artifacts/<artifact id> returns serialized Artifact

- DELETE request to the /api/v3/content/artifacts/<artifact id>/ raises an exception if the Artifact is still associated with any ContentUnit. The exception should provide a list of content units still associated with the Artifact. If an exception is not raised, Artifact is removed from the database and the file is removed from disk.

[0] https://github.com/pulp/pulp/blob/3.0-dev/platform/pulpcore/app/models/content.py#L72
[1] http://www.django-rest-framework.org/api-guide/parsers/#fileuploadparser
[2] https://docs.djangoproject.com/en/1.11/topics/http/file-uploads/#upload-handlers

Actions #1

Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)
Actions #2

Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)
Actions #3

Updated by amacdona@redhat.com over 6 years ago

  • Description updated (diff)

There is a small gotcha here. NamedModelViewSets come with an update method.

The normal DRF way to create a ViewSet without update would be to disinclude the update mixin. Since NamedModelViewSet uses the update mixin, it might be simpler to disable PUT/PATCH. https://stackoverflow.com/a/31450643/3154930

Actions #4

Updated by dkliban@redhat.com over 6 years ago

  • Subject changed from As an authenticated user, I can request a file ID from the server to upload a file with to As an authenticated user, I can create an Artifact by uploading a file
  • Description updated (diff)
Actions #5

Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)
Actions #6

Updated by mhrivnak over 6 years ago

  • Sprint/Milestone set to 41
Actions #7

Updated by dkliban@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
Actions #8

Updated by dkliban@redhat.com over 6 years ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp/pull/3080

I marked it as a work in progress for now. I just realized that the story lists GET parameters and my initial implementation uses POST. I will adjust it to use GET params.

Actions #9

Updated by dkliban@redhat.com over 6 years ago

  • Tags Pulp 3 Plugin Writer Alpha added
Actions #10

Updated by mhrivnak over 6 years ago

When the story says "GET parameters", I assume that means "query parameters" since they're being used with a POST request.

I appears that django and DRF have out-of-the-box support for multipart form data. Why not just use that? I think it's much more common than adding query parameters to a POST request.

Actions #11

Updated by dkliban@redhat.com over 6 years ago

I implemented the story using multipart form data. I was planning on emailing pulp-dev with that info later today.

Actions #12

Updated by bmbouter over 6 years ago

+1 to multipart form data.

Actions #13

Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)
Actions #14

Updated by jortel@redhat.com over 6 years ago

grooming for alpha.

Actions #15

Updated by jortel@redhat.com over 6 years ago

  • Groomed changed from No to Yes
Actions #16

Updated by dkliban@redhat.com over 6 years ago

Talked with jortel earlier today and we determined that an Artifact requires a file to exist on disk. Content created with a deferred download policy would have ContentArtifacts that don't have the Artifact foreign key set.

Actions #17

Updated by mhrivnak over 6 years ago

  • Priority changed from Normal to High
Actions #18

Updated by mhrivnak over 6 years ago

  • Sprint/Milestone changed from 41 to 42

Added by dkliban@redhat.com over 6 years ago

Revision 884ae9d7 | View on GitHub

Problem: Artifacts can't be created via REST API

Solution: add a viewset for Artifacts

The REST API user can now POST to /api/v3/artifacts endpoint with a file, size, and any number of checksums. During the upload of the file, each possible checksum is calculated. If user provides size or any checksums, they are verified after the upload finishes. If one of the values does not match a ValidationError is raised.

The user can also list all the the Artifacts by performing a GET request to /api/v3/artifacts. An Artifact can be deleted by making a DELETE request to /api/v3/artifacts/.

The user can not update Artifacts via REST API after one is created.

The 'django.core.files.uploadhandler.MemoryFileUploadHandler' is disabled so all uploads are handled by 'pulpcore.app.upload.PulpFileUploadHandler' and are written to '/var/lib/pulp/tmp' as the file is being uploaded.

closes #2843 https://pulp.plan.io/issues/2843

Added by dkliban@redhat.com over 6 years ago

Revision 884ae9d7 | View on GitHub

Problem: Artifacts can't be created via REST API

Solution: add a viewset for Artifacts

The REST API user can now POST to /api/v3/artifacts endpoint with a file, size, and any number of checksums. During the upload of the file, each possible checksum is calculated. If user provides size or any checksums, they are verified after the upload finishes. If one of the values does not match a ValidationError is raised.

The user can also list all the the Artifacts by performing a GET request to /api/v3/artifacts. An Artifact can be deleted by making a DELETE request to /api/v3/artifacts/.

The user can not update Artifacts via REST API after one is created.

The 'django.core.files.uploadhandler.MemoryFileUploadHandler' is disabled so all uploads are handled by 'pulpcore.app.upload.PulpFileUploadHandler' and are written to '/var/lib/pulp/tmp' as the file is being uploaded.

closes #2843 https://pulp.plan.io/issues/2843

Actions #19

Updated by dkliban@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #21

Updated by bmbouter about 6 years ago

  • Tags deleted (Pulp 3 Plugin Writer Alpha)

Cleaning up Redmine tags

Actions #22

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 23
Actions #23

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (42)
Actions #24

Updated by daviddavis almost 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #25

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3)
Actions #26

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF