Project

Profile

Help

Issue #4333

repo syncs fail causing worker to crash with /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standard

Added by jsherril@redhat.com 5 months ago. Updated 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Severity:
2. Medium
Version:
Platform Release:
2.18.1
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 48

Description

I'm not sure if it is all repos syncs or just some, but in Katello we see this 100% of the time with our capsules syncing from the main pulp server. /var/log/messages prints:

/usr/bin/python: symbol lookup error: /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standard

And then a long traceback where the worker crashes:

celery.app.trace:ERROR: [dd28dcb8] (3955-10304) Task pulp.server.async.tasks._release_resource[dd28dcb8-b5f3-490a-ba7c-a8fe1d4dd1d4] raised unexpected: AttributeError("'NoneType' object has no attribute 'top'",)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304) Traceback (most recent call last):
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     return super(PulpTask, self).__call__(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     return self.run(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     if not self.request.called_directly:
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     req = self.request_stack.top
celery.app.trace:ERROR: [dd28dcb8] (3955-10304) AttributeError: 'NoneType' object has no attribute 'top'
celery.worker.request:ERROR: (3604-10304) Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 127.',)
celery.worker.request:ERROR: (3604-10304) Traceback (most recent call last):
celery.worker.request:ERROR: (3604-10304)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
celery.worker.request:ERROR: (3604-10304)     human_status(exitcode)),
celery.worker.request:ERROR: (3604-10304) WorkerLostError: Worker exited prematurely: exitcode 127.

Pulp should likely handle this task error better and not crash the worker as well.


Related issues

Duplicates Pulp - Issue #4263: Tasks Stuck in Running CLOSED - DUPLICATE Actions

History

#1 Updated by jsherril@redhat.com 5 months ago

I forgot to mention this occurs with the pulp-provided libmodulemd 1.6.3, but 1.7.0 (from koji.fedorahosted) works fine.

#2 Updated by dalley 5 months ago

#4 Updated by dalley 5 months ago

  • Related to deleted (Issue #4263: Tasks Stuck in Running)

#5 Updated by dalley 5 months ago

#6 Updated by dalley 5 months ago

  • Triaged changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 47

We should update Pulp's libmodulemd dependency to 1.7.0 from https://kojipkgs.fedoraproject.org//packages/libmodulemd/1.7.0/1.el7/x86_64/

#7 Updated by dalley 5 months ago

  • Priority changed from Normal to High

#8 Updated by dalley 5 months ago

  • Blocked by Issue #4337: Update vendored libmodulemd to version 1.7.0 added

#9 Updated by dalley 5 months ago

  • Status changed from NEW to POST

#10 Updated by rchan 5 months ago

  • Sprint changed from Sprint 47 to Sprint 48

#11 Updated by dalley 5 months ago

  • Status changed from POST to MODIFIED
  • Platform Release set to 2.18.1

#12 Updated by dalley 5 months ago

  • Blocked by deleted (Issue #4337: Update vendored libmodulemd to version 1.7.0)

#13 Updated by dalley 5 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#14 Updated by jortel@redhat.com 5 months ago

  • Status changed from CLOSED - CURRENTRELEASE to ON_QA

#15 Updated by jortel@redhat.com 4 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#16 Updated by bmbouter 2 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF