Project

Profile

Help

Issue #3896

pulp_docker importer discards relevant exception details

Added by rmcgover over 1 year ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.19.0
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 46

Description

We recently hit an exception on our Pulp installation with pulp_docker:

{
 "exception": null, 
 "task_type": "pulp.server.managers.content.upload.import_uploaded_unit", 
 "_href": "/pulp/api/v2/tasks/7f3e683b-0e29-4c6e-a600-b962932a0bb0/", 
 "task_id": "7f3e683b-0e29-4c6e-a600-b962932a0bb0", 
 "tags": [
  "pulp:repository:redhat-rhel", 
  "pulp:action:import_upload" 
 ], 
 "finish_time": "2018-08-01T17:44:10Z", 
 "_ns": "task_status", 
 "start_time": "2018-08-01T17:44:10Z", 
 "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 438, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\", line 223, in import_uploaded_unit\n    unit_type=unit_type_id, summary=result['summary'], details=result['details']\nPulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository redhat-rhel.\n", 
 "spawned_tasks": [], 
 "progress_report": {}, 
 "state": "error", 
 "worker_name": "reserved_resource_worker-3@brew-pulp-docker02.web.prod.ext.phx2.redhat.com", 
 "result": null, 
 "error": {
  "code": "PLP0047", 
  "data": {
   "unit_type": "docker_manifest_list", 
   "importer_id": "docker_importer", 
   "repo_id": "redhat-rhel", 
   "details": {}, 
   "summary": "" 
  }, 
  "description": "The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository redhat-rhel.", 
  "sub_errors": []
 }, 
 "_id": {
  "$oid": "5b61f151e95a3bea68e6ef47" 
 }, 
 "id": "5b61f151e95a3bea68e6ef47" 
}

With formatted traceback:

    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
        R = retval = fun(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
        return super(Task, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
        return super(PulpTask, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
        return self.run(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 223, in import_uploaded_unit
        unit_type=unit_type_id, summary=result['summary'], details=result['details']
    PulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository redhat-rhel.

Note that the error reported via the API has empty summary & details, the traceback doesn't point at the source of the error, and the system logs contain no useful information.

Looking through the code, it seems this unhelpful error handling is introduced here in plugins/pulp_docker/plugins/importers/importer.py, upload_unit method:

        try:
            upload_step = upload.UploadStep(repo=repo, file_path=file_path, config=config,
                                            metadata=metadata, type_id=type_id)
            upload_step.process_lifecycle()
        except PulpCodedValidationException:
            raise
        except Exception as e:
            return {'success_flag': False, 'summary': e.message, 'details': {}}

Consider if an exception is raised during UploadStep constructor or process_lifecycle. Then Pulp discards all information about the exception except for the "message" (which can be empty as not all types of exceptions use a message). Thus it's virtually impossible to figure out what failed. Can it please be updated to not discard critical error information?

Actual behavior: when an exception other than PulpCodedValidationException occurs during UploadStep on docker imports, all information about the exception is discarded other than the message.

Expected behavior: when an exception occurs during UploadStep, the exception details are available either through logs or through API, including at least the type of the exception and the traceback.

Checked as of pulp_docker commit fd8d4e767ba9ca0.


Related issues

Copied to Pulp - Test #4300: pulp_docker importer discards relevant exception details NEW Actions

Associated revisions

Revision 0d775263 View on GitHub
Added by jortel@redhat.com 11 months ago

Include summary and details in PLP0047 upload failures.
closes #3896

History

#1 Updated by dkliban@redhat.com over 1 year ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 41

#2 Updated by rchan about 1 year ago

  • Sprint changed from Sprint 41 to Sprint 42

#3 Updated by rchan about 1 year ago

  • Sprint changed from Sprint 42 to Sprint 43

#4 Updated by amacdona@redhat.com about 1 year ago

  • Sprint changed from Sprint 43 to Sprint 44

#5 Updated by daviddavis about 1 year ago

  • Sprint changed from Sprint 44 to Sprint 45

#6 Updated by ipanova@redhat.com about 1 year ago

  • Project changed from Docker Support to Pulp
  • Priority changed from Normal to High

same isssue was manifested during RPM upload. this looks like a core issue and not the plugin one.

Nov 06 14:20:54 r76.pulp.vm pulp[10835]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[024d1a9e-c6d9-4d5f-b0c7-a4882e148592]
Nov 06 14:20:54 r76.pulp.vm pulp[11033]: celery.app.trace:INFO: [c0594d78] Task pulp.server.async.tasks._queue_reserved_task[c0594d78-f0e2-4c7b-9372-f9e75f78b8f3] succeeded in 0.119898567s: None
Nov 06 14:20:54 r76.pulp.vm pulp[10938]: pulp.server.async.tasks:INFO: [53955228] Task failed : [53955228-dc8e-4935-ab96-ef59c5504ad6] : Uploaded file is not a valid RPM.
Nov 06 14:20:54 r76.pulp.vm pulp[10938]: celery.app.trace:INFO: [53955228] Task pulp.server.managers.content.upload.import_uploaded_unit[53955228-dc8e-4935-ab96-ef59c5504ad6] raised expected: PulpCodedException()
Nov 06 14:20:54 r76.pulp.vm pulp[10938]: celery.app.trace:INFO: [024d1a9e] Task pulp.server.async.tasks._release_resource[024d1a9e-c6d9-4d5f-b0c7-a4882e148592] succeeded in 0.00247618999992s: None
Nov 06 14:21:47 r76.pulp.vm yum[11463]: Installed: fpaste-0.3.7.4.1-2.el7.noarch

I am also bumping the priority because this bug is annoying and makes it hard to debug what went wrong.

#7 Updated by rchan 12 months ago

  • Sprint changed from Sprint 45 to Sprint 46

#8 Updated by jortel@redhat.com 11 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#9 Updated by jortel@redhat.com 11 months ago

  • Status changed from ASSIGNED to POST

#10 Updated by jortel@redhat.com 11 months ago

  • Status changed from POST to MODIFIED

#11 Updated by bherring 10 months ago

  • Copied to Test #4300: pulp_docker importer discards relevant exception details added

#12 Updated by jortel@redhat.com 10 months ago

The error can be reproduced by uploading an archive with a sabotaged manifest. Here is an example of a manifest with a missing digest.

diff -c busybox4/manifest.json busybox/manifest.json
*** busybox4/manifest.json    2019-01-07 11:10:26.312155090 -0600
--- busybox/manifest.json    2019-01-07 12:45:17.283061706 -0600
***************
*** 10,16 ****
        {
           "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
           "size": 723146,
-          "digest": "sha256:07a152489297fc2bca20be96fab3527ceac5668328a30fd543a160cd689ee548" 
        }
     ]
! }
\ No newline at end of file
--- 10,15 ----
        {
           "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
           "size": 723146,
        }
     ]
! }

Result:

Task Failed

The importer docker_importer indicated a failed response when uploading
docker_manifest unit to repository test. Summary: '[Errno 2] No such file or
directory:
u'/var/cache/pulp/reserved_resource_worker-0@f26c.redhat.com/992e799e-d5ba-45da-
9f48-9cb30bf34f07/manifest.json''. Details: '{}' 

Deleting the upload request...
... completed

#13 Updated by ttereshc 8 months ago

  • Platform Release set to 2.19.0

#14 Updated by ttereshc 8 months ago

  • Sprint/Milestone set to 2.19.0

#15 Updated by ttereshc 8 months ago

  • Status changed from MODIFIED to ON_QA

#16 Updated by ttereshc 8 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#17 Updated by bmbouter 7 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF