Project

Profile

Help

Issue #4333

closed

repo syncs fail causing worker to crash with /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standard

Added by jsherril@redhat.com over 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.18.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 48
Quarter:

Description

I'm not sure if it is all repos syncs or just some, but in Katello we see this 100% of the time with our capsules syncing from the main pulp server. /var/log/messages prints:

/usr/bin/python: symbol lookup error: /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standard  

And then a long traceback where the worker crashes:

celery.app.trace:ERROR: [dd28dcb8] (3955-10304) Task pulp.server.async.tasks._release_resource[dd28dcb8-b5f3-490a-ba7c-a8fe1d4dd1d4] raised unexpected: AttributeError("'NoneType' object has no attribute 'top'",)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304) Traceback (most recent call last):
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     return super(PulpTask, self).__call__(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     return self.run(*args, **kwargs)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     if not self.request.called_directly:
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
celery.app.trace:ERROR: [dd28dcb8] (3955-10304)     req = self.request_stack.top
celery.app.trace:ERROR: [dd28dcb8] (3955-10304) AttributeError: 'NoneType' object has no attribute 'top'
celery.worker.request:ERROR: (3604-10304) Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 127.',)
celery.worker.request:ERROR: (3604-10304) Traceback (most recent call last):
celery.worker.request:ERROR: (3604-10304)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
celery.worker.request:ERROR: (3604-10304)     human_status(exitcode)),
celery.worker.request:ERROR: (3604-10304) WorkerLostError: Worker exited prematurely: exitcode 127.

Pulp should likely handle this task error better and not crash the worker as well.


Related issues

Is duplicate of Pulp - Issue #4263: Tasks Stuck in RunningCLOSED - DUPLICATEActions
Actions #1

Updated by jsherril@redhat.com over 5 years ago

I forgot to mention this occurs with the pulp-provided libmodulemd 1.6.3, but 1.7.0 (from koji.fedorahosted) works fine.

Actions #2

Updated by dalley about 5 years ago

Actions #4

Updated by dalley about 5 years ago

  • Related to deleted (Issue #4263: Tasks Stuck in Running)
Actions #5

Updated by dalley about 5 years ago

  • Is duplicate of Issue #4263: Tasks Stuck in Running added
Actions #6

Updated by dalley about 5 years ago

  • Triaged changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 47

We should update Pulp's libmodulemd dependency to 1.7.0 from https://kojipkgs.fedoraproject.org//packages/libmodulemd/1.7.0/1.el7/x86_64/

Actions #7

Updated by dalley about 5 years ago

  • Priority changed from Normal to High
Actions #8

Updated by dalley about 5 years ago

  • Blocked by Issue #4337: Update vendored libmodulemd to version 1.7.0 added
Actions #9

Updated by dalley about 5 years ago

  • Status changed from NEW to POST
Actions #10

Updated by rchan about 5 years ago

  • Sprint changed from Sprint 47 to Sprint 48
Actions #11

Updated by dalley about 5 years ago

  • Status changed from POST to MODIFIED
  • Platform Release set to 2.18.1
Actions #12

Updated by dalley about 5 years ago

  • Blocked by deleted (Issue #4337: Update vendored libmodulemd to version 1.7.0)
Actions #13

Updated by dalley about 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #14

Updated by jortel@redhat.com about 5 years ago

  • Status changed from CLOSED - CURRENTRELEASE to 5
Actions #15

Updated by jortel@redhat.com about 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #16

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF