Issue #3896
closedpulp_docker importer discards relevant exception details
Description
We recently hit an exception on our Pulp installation with pulp_docker:
{
"exception": null,
"task_type": "pulp.server.managers.content.upload.import_uploaded_unit",
"_href": "/pulp/api/v2/tasks/7f3e683b-0e29-4c6e-a600-b962932a0bb0/",
"task_id": "7f3e683b-0e29-4c6e-a600-b962932a0bb0",
"tags": [
"pulp:repository:redhat-rhel",
"pulp:action:import_upload"
],
"finish_time": "2018-08-01T17:44:10Z",
"_ns": "task_status",
"start_time": "2018-08-01T17:44:10Z",
"traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n return super(Task, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n return super(PulpTask, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 438, in __protected_call__\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\", line 223, in import_uploaded_unit\n unit_type=unit_type_id, summary=result['summary'], details=result['details']\nPulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository redhat-rhel.\n",
"spawned_tasks": [],
"progress_report": {},
"state": "error",
"worker_name": "reserved_resource_worker-3@brew-pulp-docker02.web.prod.ext.phx2.redhat.com",
"result": null,
"error": {
"code": "PLP0047",
"data": {
"unit_type": "docker_manifest_list",
"importer_id": "docker_importer",
"repo_id": "redhat-rhel",
"details": {},
"summary": ""
},
"description": "The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository redhat-rhel.",
"sub_errors": []
},
"_id": {
"$oid": "5b61f151e95a3bea68e6ef47"
},
"id": "5b61f151e95a3bea68e6ef47"
}
With formatted traceback:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
return super(Task, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
return super(PulpTask, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 223, in import_uploaded_unit
unit_type=unit_type_id, summary=result['summary'], details=result['details']
PulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository redhat-rhel.
Note that the error reported via the API has empty summary & details, the traceback doesn't point at the source of the error, and the system logs contain no useful information.
Looking through the code, it seems this unhelpful error handling is introduced here in plugins/pulp_docker/plugins/importers/importer.py, upload_unit method:
try:
upload_step = upload.UploadStep(repo=repo, file_path=file_path, config=config,
metadata=metadata, type_id=type_id)
upload_step.process_lifecycle()
except PulpCodedValidationException:
raise
except Exception as e:
return {'success_flag': False, 'summary': e.message, 'details': {}}
Consider if an exception is raised during UploadStep constructor or process_lifecycle. Then Pulp discards all information about the exception except for the "message" (which can be empty as not all types of exceptions use a message). Thus it's virtually impossible to figure out what failed. Can it please be updated to not discard critical error information?
Actual behavior: when an exception other than PulpCodedValidationException occurs during UploadStep on docker imports, all information about the exception is discarded other than the message.
Expected behavior: when an exception occurs during UploadStep, the exception details are available either through logs or through API, including at least the type of the exception and the traceback.
Checked as of pulp_docker commit fd8d4e767ba9ca0.
Related issues
Updated by dkliban@redhat.com about 6 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 41
Updated by amacdona@redhat.com almost 6 years ago
- Sprint changed from Sprint 43 to Sprint 44
Updated by daviddavis almost 6 years ago
- Sprint changed from Sprint 44 to Sprint 45
Updated by ipanova@redhat.com almost 6 years ago
- Project changed from Docker Support to Pulp
- Priority changed from Normal to High
same isssue was manifested during RPM upload. this looks like a core issue and not the plugin one.
Nov 06 14:20:54 r76.pulp.vm pulp[10835]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[024d1a9e-c6d9-4d5f-b0c7-a4882e148592]
Nov 06 14:20:54 r76.pulp.vm pulp[11033]: celery.app.trace:INFO: [c0594d78] Task pulp.server.async.tasks._queue_reserved_task[c0594d78-f0e2-4c7b-9372-f9e75f78b8f3] succeeded in 0.119898567s: None
Nov 06 14:20:54 r76.pulp.vm pulp[10938]: pulp.server.async.tasks:INFO: [53955228] Task failed : [53955228-dc8e-4935-ab96-ef59c5504ad6] : Uploaded file is not a valid RPM.
Nov 06 14:20:54 r76.pulp.vm pulp[10938]: celery.app.trace:INFO: [53955228] Task pulp.server.managers.content.upload.import_uploaded_unit[53955228-dc8e-4935-ab96-ef59c5504ad6] raised expected: PulpCodedException()
Nov 06 14:20:54 r76.pulp.vm pulp[10938]: celery.app.trace:INFO: [024d1a9e] Task pulp.server.async.tasks._release_resource[024d1a9e-c6d9-4d5f-b0c7-a4882e148592] succeeded in 0.00247618999992s: None
Nov 06 14:21:47 r76.pulp.vm yum[11463]: Installed: fpaste-0.3.7.4.1-2.el7.noarch
I am also bumping the priority because this bug is annoying and makes it hard to debug what went wrong.
Updated by jortel@redhat.com over 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to jortel@redhat.com
Added by jortel@redhat.com over 5 years ago
Updated by jortel@redhat.com over 5 years ago
- Status changed from ASSIGNED to POST
Updated by jortel@redhat.com over 5 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp|0d7752637c33d5391f4b5ddba465e6448e5279c4.
Updated by bherring over 5 years ago
- Copied to Test #4300: pulp_docker importer discards relevant exception details added
Updated by jortel@redhat.com over 5 years ago
The error can be reproduced by uploading an archive with a sabotaged manifest. Here is an example of a manifest with a missing digest.
diff -c busybox4/manifest.json busybox/manifest.json
*** busybox4/manifest.json 2019-01-07 11:10:26.312155090 -0600
--- busybox/manifest.json 2019-01-07 12:45:17.283061706 -0600
***************
*** 10,16 ****
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 723146,
- "digest": "sha256:07a152489297fc2bca20be96fab3527ceac5668328a30fd543a160cd689ee548"
}
]
! }
\ No newline at end of file
--- 10,15 ----
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 723146,
}
]
! }
Result:
Task Failed
The importer docker_importer indicated a failed response when uploading
docker_manifest unit to repository test. Summary: '[Errno 2] No such file or
directory:
u'/var/cache/pulp/reserved_resource_worker-0@f26c.redhat.com/992e799e-d5ba-45da-
9f48-9cb30bf34f07/manifest.json''. Details: '{}'
Deleting the upload request...
... completed
Updated by ttereshc over 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Include summary and details in PLP0047 upload failures. closes #3896