Project

Profile

Help

Story #2843

As an authenticated user, I can create an Artifact by uploading a file

Added by dkliban@redhat.com over 2 years ago. Updated 8 months ago.

Status:
MODIFIED
Priority:
High
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 23

Description

For an API user to create an Artifact, Pulp 3 needs to have the following:

- updated Artifact0 model without a 'content' foreign key nor the 'relative_path' field. A uniqueness constraint should be added on the sha256 field.
- viewset that can handle CRD operations for Artifacts. It most likely needs to use the FileUploadParser1 and a custom Django Upload Handler2.
- serializer for the viewset which will return all serialized fields of the Artifact model. The file field should be it's full path on disk.
- API endpoint at /api/v3/content/artifacts
- POST request to the /api/v3/content/artifacts/ endpoint creates an Artifact. The body of the request contains multipart form data with the following:

file - The file being uploaded
size - The size of the file in bytes.
md5 - The MD5 checksum of the file.
sha1 - The SHA-1 checksum of the file.
sha224 - The SHA-224 checksum of the file.
sha256 - The SHA-256 checksum of the file.
sha384 - The SHA-384 checksum of the file.
sha512 - The SHA-512 checksum of the file.

Before the model is saved, a SHA-256 checksum (digest) of the uploaded file is generated.
If a sha256 was specified in POST parameters, the generated hash is validated against the value specified as the POST parameter. If the values don't match a validation exception is raised.
If an Artifact with the same sha256 checksum already exists, a 400 response is returned to the user.
When the model is saved, the file is written to MEDIA_ROOT/content/units/digest[0:2]/digest[2:]
After a successful save, a serialized version of the Artifact is returned.

- GET request to the /api/v3/content/artifacts/<artifact id> returns serialized Artifact

- DELETE request to the /api/v3/content/artifacts/<artifact id>/ raises an exception if the Artifact is still associated with any ContentUnit. The exception should provide a list of content units still associated with the Artifact. If an exception is not raised, Artifact is removed from the database and the file is removed from disk.

[0] https://github.com/pulp/pulp/blob/3.0-dev/platform/pulpcore/app/models/content.py#L72
[1] http://www.django-rest-framework.org/api-guide/parsers/#fileuploadparser
[2] https://docs.djangoproject.com/en/1.11/topics/http/file-uploads/#upload-handlers

Associated revisions

Revision 884ae9d7 View on GitHub
Added by dkliban@redhat.com over 2 years ago

Problem: Artifacts can't be created via REST API

Solution: add a viewset for Artifacts

The REST API user can now POST to /api/v3/artifacts endpoint with a file,
size, and any number of checksums. During the upload of the file, each
possible checksum is calculated. If user provides size or any checksums,
they are verified after the upload finishes. If one of the values does not
match a ValidationError is raised.

The user can also list all the the Artifacts by performing a GET request
to /api/v3/artifacts. An Artifact can be deleted by making a DELETE
request to /api/v3/artifacts/<pk>.

The user can not update Artifacts via REST API after one is created.

The 'django.core.files.uploadhandler.MemoryFileUploadHandler' is disabled so
all uploads are handled by 'pulpcore.app.upload.PulpFileUploadHandler' and
are written to '/var/lib/pulp/tmp' as the file is being uploaded.

closes #2843
https://pulp.plan.io/issues/2843

Revision 884ae9d7 View on GitHub
Added by dkliban@redhat.com over 2 years ago

Problem: Artifacts can't be created via REST API

Solution: add a viewset for Artifacts

The REST API user can now POST to /api/v3/artifacts endpoint with a file,
size, and any number of checksums. During the upload of the file, each
possible checksum is calculated. If user provides size or any checksums,
they are verified after the upload finishes. If one of the values does not
match a ValidationError is raised.

The user can also list all the the Artifacts by performing a GET request
to /api/v3/artifacts. An Artifact can be deleted by making a DELETE
request to /api/v3/artifacts/<pk>.

The user can not update Artifacts via REST API after one is created.

The 'django.core.files.uploadhandler.MemoryFileUploadHandler' is disabled so
all uploads are handled by 'pulpcore.app.upload.PulpFileUploadHandler' and
are written to '/var/lib/pulp/tmp' as the file is being uploaded.

closes #2843
https://pulp.plan.io/issues/2843

History

#1 Updated by dkliban@redhat.com over 2 years ago

  • Description updated (diff)

#2 Updated by dkliban@redhat.com over 2 years ago

  • Description updated (diff)

#3 Updated by amacdona@redhat.com over 2 years ago

  • Description updated (diff)

There is a small gotcha here. NamedModelViewSets come with an update method.

The normal DRF way to create a ViewSet without update would be to disinclude the update mixin. Since NamedModelViewSet uses the update mixin, it might be simpler to disable PUT/PATCH. https://stackoverflow.com/a/31450643/3154930

#4 Updated by dkliban@redhat.com over 2 years ago

  • Subject changed from As an authenticated user, I can request a file ID from the server to upload a file with to As an authenticated user, I can create an Artifact by uploading a file
  • Description updated (diff)

#5 Updated by dkliban@redhat.com over 2 years ago

  • Description updated (diff)

#6 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone set to 41

#7 Updated by dkliban@redhat.com over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#8 Updated by dkliban@redhat.com over 2 years ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp/pull/3080

I marked it as a work in progress for now. I just realized that the story lists GET parameters and my initial implementation uses POST. I will adjust it to use GET params.

#9 Updated by dkliban@redhat.com over 2 years ago

  • Tags Pulp 3 Plugin Writer Alpha added

#10 Updated by mhrivnak over 2 years ago

When the story says "GET parameters", I assume that means "query parameters" since they're being used with a POST request.

I appears that django and DRF have out-of-the-box support for multipart form data. Why not just use that? I think it's much more common than adding query parameters to a POST request.

#11 Updated by dkliban@redhat.com over 2 years ago

I implemented the story using multipart form data. I was planning on emailing pulp-dev with that info later today.

#12 Updated by bmbouter over 2 years ago

+1 to multipart form data.

#13 Updated by dkliban@redhat.com over 2 years ago

  • Description updated (diff)

#14 Updated by jortel@redhat.com over 2 years ago

grooming for alpha.

#15 Updated by jortel@redhat.com over 2 years ago

  • Groomed changed from No to Yes

#16 Updated by dkliban@redhat.com over 2 years ago

Talked with jortel earlier today and we determined that an Artifact requires a file to exist on disk. Content created with a deferred download policy would have ContentArtifacts that don't have the Artifact foreign key set.

#17 Updated by mhrivnak over 2 years ago

  • Priority changed from Normal to High

#18 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone changed from 41 to 42

#19 Updated by dkliban@redhat.com over 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#20 Updated by kersom over 2 years ago

  • Smash Test set to 726

#21 Updated by bmbouter almost 2 years ago

  • Tags deleted (Pulp 3 Plugin Writer Alpha)

Cleaning up Redmine tags

#22 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 23

#23 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (42)

#24 Updated by daviddavis 8 months ago

  • Sprint/Milestone set to 3.0

#25 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF