Project

Profile

Help

Issue #9550

closed

Ensure correct tmp location is used when reconstructing chunked uploads

Added by wibbit about 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Currently when uploading content via the chunked uploads, when the file is reconstructed it uses the generic /tmp location as opposed to the are specified by either WORKING_DIRECTORY or FILE_UPLOAD_DIRECTORY.

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/upload.py#L29


Related issues

Related to Pulp - Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directoryCLOSED - DUPLICATE

Actions
Copied to Pulp - Backport #9556: Backport 9550 to 3.14: Ensure correct tmp location is used when reconstructing chunked uploadsCLOSED - CURRENTRELEASEdkliban@redhat.com

Actions
Actions #1

Updated by dkliban@redhat.com about 3 years ago

diff --git a/pulpcore/app/tasks/upload.py b/pulpcore/app/tasks/upload.py
index 89d876816..dbd140f90 100644
--- a/pulpcore/app/tasks/upload.py
+++ b/pulpcore/app/tasks/upload.py
@@ -26,7 +26,7 @@ def commit(upload_id, sha256):
         return
 
     chunks = models.UploadChunk.objects.filter(upload=upload).order_by("offset")
-    with NamedTemporaryFile("ab") as temp_file:
+    with NamedTemporaryFile(mode="ab", dir=".", delete=False) as temp_file:
         for chunk in chunks:
             temp_file.write(chunk.file.read())
         temp_file.flush()

That patch got him as far as trying to create an RPM from a file but then pulp_rpm plugin complained about the file.



        [dfurlong@snjlv-sysengutil01 ~]$ time /opt/systems/bin/syseng-pulpctlv3 --username=admin content-upload --type=rpm --platform=rhel7 --repo=acme --release=trunk --path=/home/dfurlong/acme-trino-363-1.el7.x86_64.rpm
        2021-11-02 15:12:44|ERROR   |Server: ncc1r8.us.acme.net, Task: /pulp/api/v3/tasks/452c9c7e-1715-4751-9462-0f68a649feb3/ is in a failed state. With message: RPM file cannot be parsed for metadata.
        Traceback (most recent call last):
          File "/opt/systems/bin/syseng-pulpctlv3", line 11, in <module>
            controller.run()
          File "/home/dfurlong/systems/syseng_code/python/site_syseng/prog/pulpctl/pulpctl.py", line 1107, in run
            self.args.action(self)
          File "/home/dfurlong/systems/syseng_code/python/site_syseng/prog/pulpctl/pulpctl.py", line 437, in <lambda>
            content_upload_subparser.set_defaults(action=lambda s: cls.do_content_upload(s))
          File "/home/dfurlong/systems/syseng_code/python/site_syseng/prog/pulpctl/pulpctl.py", line 1601, in do_content_upload
            self.pulp3_servers.primary.content_create(artifacts, self.args.type)
          File "/home/dfurlong/systems/syseng_code/python/site_syseng/prog/pulpctl/pulpserver.py", line 204, in content_create
            href = self.tasks_retrieve_create_resource_7708(task_href=task, type=type)
          File "/home/dfurlong/systems/syseng_code/python/site_syseng/prog/pulpctl/pulpserver.py", line 671, in tasks_retrieve_create_resource_7708
            raise ValueError(
        ValueError: ('task_href: %s, has failed with error %f', '/pulp/api/v3/tasks/452c9c7e-1715-4751-9462-0f68a649feb3/', {'traceback': '  File "/opt/acme/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task\n    result = func(*args, **kwargs)\n  File "/opt/acme/lib/python3.8/site-packages/pulpcore/app/tasks/base.py", line 34, in general_create\n    serializer.is_valid(raise_exception=True)\n  File "/opt/acme/lib/python3.8/site-packages/rest_framework/serializers.py", line 220, in is_valid\n    self._validated_data = self.run_validation(self.initial_data)\n  File "/opt/acme/lib/python3.8/site-packages/rest_framework/serializers.py", line 422, in run_validation\n    value = self.validate(value)\n  File "/opt/acme/lib/python3.8/site-packages/pulpcore/plugin/serializers/content.py", line 104, in validate\n    data = self.deferred_validate(data)\n  File "/opt/acme/lib/python3.8/site-packages/pulp_rpm/app/serializers/package.py", line 240, in deferred_validate\n    raise NotAcceptable(detail="RPM file cannot be parsed for metadata.")\n', 'description': 'RPM file cannot be parsed for metadata.'})
Actions #2

Updated by dkliban@redhat.com about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
Actions #3

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST

Added by dkliban@redhat.com about 3 years ago

Revision b7210102 | View on GitHub

Assembles chunked_uploads in workers working directory

fixes: #9550 https://pulp.plan.io/issues/9550

Actions #4

Updated by dkliban@redhat.com about 3 years ago

  • Status changed from POST to MODIFIED
Actions #5

Updated by dkliban@redhat.com about 3 years ago

  • Copied to Backport #9556: Backport 9550 to 3.14: Ensure correct tmp location is used when reconstructing chunked uploads added
Actions #6

Updated by ttereshc about 3 years ago

  • Related to Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directory added
Actions #7

Updated by pulpbot about 3 years ago

  • Sprint/Milestone set to 3.17.0
Actions #8

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF