Project

Profile

Help

Issue #4405

closed

Pulp 2.19master nightly ci regression at test_modularity.py

Added by bherring almost 6 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.19.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 49
Quarter:

Description

Failure Scenario

On nightly ci, test_modularity.py runs on 2.19master and fails in such a way that the box is no longer responsive to API testing.

Note this test works and was re-tested after the above error on 2.19 without issue on 2.18.1b1.

Information:

Failure Seen

  • Test Harness locally (RHEL76 base + Pulp 2 installed, non-FIPS) fails and dies at test_modularity
  • Jenkins also fails at the same location killing the test harness

journalctl output at time of failure

Feb 11 01:42:00 host-172-16-46-47.openstacklocal kernel: celery[12053]: segfault at 0 ip 00007f9b1e4a8b99 sp 00007ffe64b687e0 error 4 in libsolv.so.0[7f9b1e43f000+89000]
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork.
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)   "MongoClient opened before fork. Create MongoClient "
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.async.tasks:INFO: [9cbb7189] Task failed : [0349bb3b-ebb7-4585-810d-8b46ed94cb8f] : Worker terminated abnormally while processing task 03
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Task pulp.server.async.tasks._release_resource[9cbb7189-331f-42f5-b957-0d05211d5c62] raised unexpect
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Traceback (most recent call last):
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     R = retval = fun(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return super(PulpTask, self).__call__(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return self.run(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     if not self.request.called_directly:
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     req = self.request_stack.top
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) AttributeError: 'NoneType' object has no attribute 'top'
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Traceback (most recent call last):
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)     human_status(exitcode)),
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

Related issues

Blocks RPM Support - Test #4364: Implement modularity content dependency solvingCLOSED - COMPLETEbherringActions
Actions #1

Updated by bherring almost 6 years ago

Adding 2.18.1b1 vs 2.19-master diff

[root@rhel76-2181b1 ~]# diff 218b1.rpms.list  219master.rpms.list 
230,231c230,231
< libtomcrypt-1.17-26.el7.x86_64
< libtommath-0.42.0-6.el7.x86_64
---
> libtomcrypt-1.17-25.el7.x86_64
> libtommath-0.42.0-5.el7.x86_64
372,387c372,387
< pulp-admin-client-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-deb-admin-extensions-1.8.0-1.el7.noarch
< pulp-deb-plugins-1.8.0-1.el7.noarch
< pulp-docker-admin-extensions-3.2.2-1.el7.noarch
< pulp-docker-plugins-3.2.2-1.el7.noarch
< pulp-ostree-admin-extensions-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-ostree-plugins-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-admin-extensions-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-plugins-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-tools-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-python-admin-extensions-2.0.3-1.el7.noarch
< pulp-python-plugins-2.0.3-1.el7.noarch
< pulp-rpm-admin-extensions-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-rpm-plugins-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-selinux-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-server-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> pulp-admin-client-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-deb-admin-extensions-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-deb-plugins-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-docker-admin-extensions-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-docker-plugins-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-ostree-plugins-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-puppet-admin-extensions-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-plugins-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-tools-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-python-admin-extensions-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-python-plugins-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-rpm-admin-extensions-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-rpm-plugins-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-selinux-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-server-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
401c401
< python2-crypto-2.6.1-15.el7.x86_64
---
> python2-crypto-2.6.1-13.el7.x86_64
406c406
< python2-jmespath-0.9.0-3.el7.noarch
---
> python2-jmespath-0.9.0-1.el7.noarch
427c427
< python-crane-3.3.0-1.el7.noarch
---
> python-crane-3.3.0-0.1.alpha.201902110520git579a2a1.el7.noarch
443c443
< python-httplib2-0.9.2-1.el7.noarch
---
> python-httplib2-0.9.2-0.1.el7.noarch
469,480c469,480
< python-pulp-bindings-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-client-lib-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-deb-common-1.8.0-1.el7.noarch
< python-pulp-docker-common-3.2.2-1.el7.noarch
< python-pulp-oid_validation-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-ostree-common-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-puppet-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-python-common-2.0.3-1.el7.noarch
< python-pulp-repoauth-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-rpm-common-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< python-pulp-streamer-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> python-pulp-bindings-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-client-lib-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-common-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-deb-common-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> python-pulp-docker-common-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> python-pulp-oid_validation-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-ostree-common-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> python-pulp-puppet-common-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> python-pulp-python-common-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> python-pulp-repoauth-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-rpm-common-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> python-pulp-streamer-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
559c559
< sshpass-1.06-2.el7.x86_64
---
> sshpass-1.06-1.el7.x86_64
Actions #2

Updated by dalley almost 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

I reproduced this on CentOS, but could not reproduce it on Fedora, as with many other bugs recently, most notably the libmodulemd one. I'm not really sure what's up with that.

Actions #3

Updated by dalley almost 6 years ago

I'm fairly certain this is a libsolv problem. 0.7.2 (Fedora) works, 0.6.34 does not, with the same Pulp code.

Discussion with Igor:

<dalley> so I think it might actually be an already resolved libsolv issue -- it works perfectly on Fedora 29 w/ v0.7.2, but segfaults on CentOS with 0.6.34-2
<ignatenkobrain> forget about 0.6.x =)
<dalley> it's what CentOS provides :) is it something that could be safely upgraded if I pinged the maintainer, or should we just vendor it in our own repo?
<ignatenkobrain> 0.7 bumps soname because it removed few functions which were part of ABI
<ignatenkobrain> but nobody used them for last 5 years
<ignatenkobrain> so for now you can just develop code on f29 and later just decide how to proceed with EL
Actions #4

Updated by CodeHeeler almost 6 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 48
Actions #6

Updated by pcreech almost 6 years ago

After working through various builds of libsolv, I'm suggesting we update the libsolv dependency to the 0.7.x version.

I'm usure why the segfault happens, but in various builds < 0.7.0, it occured.

Actions #7

Updated by dalley almost 6 years ago

  • Assignee changed from dalley to pcreech

Added by bherring almost 6 years ago

Revision 92118dd8 | View on GitHub

Skipping modular tests due to #4405

Issue #4405 is a regression due to libsolv in RHEL releases affecting all modular tests and hanging Jenkins CI.

Skipping tests until the issue is resolved.

refs #4405

Actions #8

Updated by rchan almost 6 years ago

  • Sprint changed from Sprint 48 to Sprint 49
Actions #9

Updated by bherring almost 6 years ago

  • Blocks Test #4364: Implement modularity content dependency solving added
Actions #10

Updated by bherring almost 6 years ago

  • Status changed from ASSIGNED to MODIFIED

Update

pcreech built libsolv. Tested with the version in 2-master with #4405 tests commenting out the skip.

Tests passed without issue.

Moving to `MODIFIED`.

Actions #11

Updated by ttereshc over 5 years ago

  • Platform Release set to 2.19.0
Actions #12

Updated by ttereshc over 5 years ago

  • Sprint/Milestone set to 2.19.0
Actions #13

Updated by ttereshc over 5 years ago

  • Status changed from MODIFIED to 5
Actions #14

Updated by ttereshc over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #15

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF