Actions
Issue #4405
closed
Pulp 2.19master nightly ci regression at test_modularity.py
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.19.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 49
Quarter:
Description
Failure Scenario¶
On nightly ci, test_modularity.py runs on 2.19master and fails in such a way that the box is no longer responsive to API testing.
Note this test works and was re-tested after the above error on 2.19 without issue on 2.18.1b1.
Information:¶
- Test: https://github.com/PulpQE/Pulp-2-Tests/blob/master/pulp_2_tests/tests/rpm/api_v2/test_modularity.py
- Repeatable: Every time
- Build: 2.19
- Branch: 2-master
Failure Seen¶
- Test Harness locally (RHEL76 base + Pulp 2 installed, non-FIPS) fails and dies at test_modularity
- Jenkins also fails at the same location killing the test harness
journalctl output at time of failure¶
Feb 11 01:42:00 host-172-16-46-47.openstacklocal kernel: celery[12053]: segfault at 0 ip 00007f9b1e4a8b99 sp 00007ffe64b687e0 error 4 in libsolv.so.0[7f9b1e43f000+89000]
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork.
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032) "MongoClient opened before fork. Create MongoClient "
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.async.tasks:INFO: [9cbb7189] Task failed : [0349bb3b-ebb7-4585-810d-8b46ed94cb8f] : Worker terminated abnormally while processing task 03
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Task pulp.server.async.tasks._release_resource[9cbb7189-331f-42f5-b957-0d05211d5c62] raised unexpect
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Traceback (most recent call last):
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) R = retval = fun(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) return super(PulpTask, self).__call__(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) return self.run(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) new_task.on_failure(exception, task_id, (), {}, MyEinfo)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) if not self.request.called_directly:
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) req = self.request_stack.top
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) AttributeError: 'NoneType' object has no attribute 'top'
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Traceback (most recent call last):
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) human_status(exitcode)),
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).
Related issues
Updated by bherring about 5 years ago
Adding 2.18.1b1 vs 2.19-master diff¶
[root@rhel76-2181b1 ~]# diff 218b1.rpms.list 219master.rpms.list
230,231c230,231
< libtomcrypt-1.17-26.el7.x86_64
< libtommath-0.42.0-6.el7.x86_64
---
> libtomcrypt-1.17-25.el7.x86_64
> libtommath-0.42.0-5.el7.x86_64
372,387c372,387
< pulp-admin-client-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-deb-admin-extensions-1.8.0-1.el7.noarch
< pulp-deb-plugins-1.8.0-1.el7.noarch
< pulp-docker-admin-extensions-3.2.2-1.el7.noarch
< pulp-docker-plugins-3.2.2-1.el7.noarch
< pulp-ostree-admin-extensions-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-ostree-plugins-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-admin-extensions-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-plugins-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-tools-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-python-admin-extensions-2.0.3-1.el7.noarch
< pulp-python-plugins-2.0.3-1.el7.noarch
< pulp-rpm-admin-extensions-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-rpm-plugins-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-selinux-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-server-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> pulp-admin-client-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-deb-admin-extensions-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-deb-plugins-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-docker-admin-extensions-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-docker-plugins-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-ostree-plugins-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-puppet-admin-extensions-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-plugins-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-tools-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-python-admin-extensions-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-python-plugins-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-rpm-admin-extensions-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-rpm-plugins-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-selinux-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-server-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
401c401
< python2-crypto-2.6.1-15.el7.x86_64
---
> python2-crypto-2.6.1-13.el7.x86_64
406c406
< python2-jmespath-0.9.0-3.el7.noarch
---
> python2-jmespath-0.9.0-1.el7.noarch
427c427
< python-crane-3.3.0-1.el7.noarch
---
> python-crane-3.3.0-0.1.alpha.201902110520git579a2a1.el7.noarch
443c443
< python-httplib2-0.9.2-1.el7.noarch
---
> python-httplib2-0.9.2-0.1.el7.noarch
469,480c469,480
< python-pulp-bindings-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-client-lib-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-deb-common-1.8.0-1.el7.noarch
< python-pulp-docker-common-3.2.2-1.el7.noarch
< python-pulp-oid_validation-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-ostree-common-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-puppet-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-python-common-2.0.3-1.el7.noarch
< python-pulp-repoauth-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-rpm-common-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< python-pulp-streamer-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> python-pulp-bindings-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-client-lib-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-common-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-deb-common-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> python-pulp-docker-common-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> python-pulp-oid_validation-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-ostree-common-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> python-pulp-puppet-common-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> python-pulp-python-common-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> python-pulp-repoauth-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-rpm-common-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> python-pulp-streamer-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
559c559
< sshpass-1.06-2.el7.x86_64
---
> sshpass-1.06-1.el7.x86_64
Updated by dalley about 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
I reproduced this on CentOS, but could not reproduce it on Fedora, as with many other bugs recently, most notably the libmodulemd one. I'm not really sure what's up with that.
Updated by dalley about 5 years ago
I'm fairly certain this is a libsolv problem. 0.7.2 (Fedora) works, 0.6.34 does not, with the same Pulp code.
Discussion with Igor:
<dalley> so I think it might actually be an already resolved libsolv issue -- it works perfectly on Fedora 29 w/ v0.7.2, but segfaults on CentOS with 0.6.34-2
<ignatenkobrain> forget about 0.6.x =)
<dalley> it's what CentOS provides :) is it something that could be safely upgraded if I pinged the maintainer, or should we just vendor it in our own repo?
<ignatenkobrain> 0.7 bumps soname because it removed few functions which were part of ABI
<ignatenkobrain> but nobody used them for last 5 years
<ignatenkobrain> so for now you can just develop code on f29 and later just decide how to proceed with EL
Updated by CodeHeeler about 5 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 48
Updated by pcreech about 5 years ago
After working through various builds of libsolv, I'm suggesting we update the libsolv dependency to the 0.7.x version.
I'm usure why the segfault happens, but in various builds < 0.7.0, it occured.
Added by bherring about 5 years ago
Updated by bherring about 5 years ago
- Blocks Test #4364: Implement modularity content dependency solving added
Updated by ttereshc about 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
.
Skipping modular tests due to #4405
Issue #4405 is a regression due to libsolv in RHEL releases affecting all modular tests and hanging Jenkins CI.
Skipping tests until the issue is resolved.
refs #4405