Project

Profile

Help

Issue #4333

closed

repo syncs fail causing worker to crash with /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standard

Added by jsherril@redhat.com about 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.18.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 48
Quarter:

Description

I'm not sure if it is all repos syncs or just some, but in Katello we see this 100% of the time with our capsules syncing from the main pulp server. /var/log/messages prints:

/usr/bin/python: symbol lookup error: /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standard  

And then a long traceback where the worker crashes:

celery.app.trace:ERROR: [dd28dcb8] (3955-10304) Task pulp.server.async.tasks._release_resource[dd28dcb8-b5f3-490a-ba7c-a8fe1d4dd1d4] raised unexpected: AttributeError("'NoneType' object has no attribute 'top'",)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304) Traceback (most recent call last):
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     return super(PulpTask, self).__call__(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     return self.run(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     if not self.request.called_directly:
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     req = self.request_stack.top
celery.app.trace:ERROR: [dd28dcb8] (3955-10304) AttributeError: 'NoneType' object has no attribute 'top'
celery.worker.request:ERROR: (3604-10304) Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 127.',)
celery.worker.request:ERROR: (3604-10304) Traceback (most recent call last):
celery.worker.request:ERROR: (3604-10304)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
celery.worker.request:ERROR: (3604-10304)     human_status(exitcode)),
celery.worker.request:ERROR: (3604-10304) WorkerLostError: Worker exited prematurely: exitcode 127.

Pulp should likely handle this task error better and not crash the worker as well.


Related issues

Is duplicate of Pulp - Issue #4263: Tasks Stuck in RunningCLOSED - DUPLICATEActions

Also available in: Atom PDF