Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #4405

closed

Pulp 2.19master nightly ci regression at test_modularity.py

Added by bherring about 5 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
pcreech
Category:
-
Sprint/Milestone:
2.19.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.19.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 49
Quarter:

Description

Failure Scenario

On nightly ci, test_modularity.py runs on 2.19master and fails in such a way that the box is no longer responsive to API testing.

Note this test works and was re-tested after the above error on 2.19 without issue on 2.18.1b1.

Information:

Failure Seen

  • Test Harness locally (RHEL76 base + Pulp 2 installed, non-FIPS) fails and dies at test_modularity
  • Jenkins also fails at the same location killing the test harness

journalctl output at time of failure

Feb 11 01:42:00 host-172-16-46-47.openstacklocal kernel: celery[12053]: segfault at 0 ip 00007f9b1e4a8b99 sp 00007ffe64b687e0 error 4 in libsolv.so.0[7f9b1e43f000+89000]
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork.
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)   "MongoClient opened before fork. Create MongoClient "
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.async.tasks:INFO: [9cbb7189] Task failed : [0349bb3b-ebb7-4585-810d-8b46ed94cb8f] : Worker terminated abnormally while processing task 03
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Task pulp.server.async.tasks._release_resource[9cbb7189-331f-42f5-b957-0d05211d5c62] raised unexpect
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Traceback (most recent call last):
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     R = retval = fun(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return super(PulpTask, self).__call__(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return self.run(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     if not self.request.called_directly:
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     req = self.request_stack.top
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) AttributeError: 'NoneType' object has no attribute 'top'
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Traceback (most recent call last):
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)     human_status(exitcode)),
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

Related issues

Blocks RPM Support - Test #4364: Implement modularity content dependency solvingCLOSED - COMPLETEbherringActions
Actions
Send by e-mail Copy link

Also available in: Atom PDF