Project

Profile

Help

Issue #4405

Pulp 2.19master nightly ci regression at test_modularity.py

Added by bherring 12 months ago. Updated 10 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.19.0
Blocks Release:
2.19.z
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 49

Description

Failure Scenario

On nightly ci, test_modularity.py runs on 2.19master and fails in such a way that the box is no longer responsive to API testing.

Note this test works and was re-tested after the above error on 2.19 without issue on 2.18.1b1.

Information:

Failure Seen

  • Test Harness locally (RHEL76 base + Pulp 2 installed, non-FIPS) fails and dies at test_modularity
  • Jenkins also fails at the same location killing the test harness

journalctl output at time of failure

Feb 11 01:42:00 host-172-16-46-47.openstacklocal kernel: celery[12053]: segfault at 0 ip 00007f9b1e4a8b99 sp 00007ffe64b687e0 error 4 in libsolv.so.0[7f9b1e43f000+89000]
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork.
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)   "MongoClient opened before fork. Create MongoClient " 
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.async.tasks:INFO: [9cbb7189] Task failed : [0349bb3b-ebb7-4585-810d-8b46ed94cb8f] : Worker terminated abnormally while processing task 03
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Task pulp.server.async.tasks._release_resource[9cbb7189-331f-42f5-b957-0d05211d5c62] raised unexpect
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Traceback (most recent call last):
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     R = retval = fun(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return super(PulpTask, self).__call__(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return self.run(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     if not self.request.called_directly:
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     req = self.request_stack.top
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) AttributeError: 'NoneType' object has no attribute 'top'
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Traceback (most recent call last):
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)     human_status(exitcode)),
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

Related issues

Blocks RPM Support - Test #4364: Implement modularity content dependency solving CLOSED - COMPLETE Actions

Associated revisions

Revision 92118dd8 View on GitHub
Added by bherring 12 months ago

Skipping modular tests due to #4405

Issue #4405 is a regression due to libsolv in RHEL releases
affecting all modular tests and hanging Jenkins CI.

Skipping tests until the issue is resolved.

refs #4405

History

#1 Updated by bherring 12 months ago

Adding 2.18.1b1 vs 2.19-master diff

[root@rhel76-2181b1 ~]# diff 218b1.rpms.list  219master.rpms.list 
230,231c230,231
< libtomcrypt-1.17-26.el7.x86_64
< libtommath-0.42.0-6.el7.x86_64
---
> libtomcrypt-1.17-25.el7.x86_64
> libtommath-0.42.0-5.el7.x86_64
372,387c372,387
< pulp-admin-client-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-deb-admin-extensions-1.8.0-1.el7.noarch
< pulp-deb-plugins-1.8.0-1.el7.noarch
< pulp-docker-admin-extensions-3.2.2-1.el7.noarch
< pulp-docker-plugins-3.2.2-1.el7.noarch
< pulp-ostree-admin-extensions-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-ostree-plugins-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-admin-extensions-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-plugins-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-tools-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-python-admin-extensions-2.0.3-1.el7.noarch
< pulp-python-plugins-2.0.3-1.el7.noarch
< pulp-rpm-admin-extensions-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-rpm-plugins-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-selinux-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-server-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> pulp-admin-client-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-deb-admin-extensions-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-deb-plugins-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-docker-admin-extensions-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-docker-plugins-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-ostree-plugins-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-puppet-admin-extensions-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-plugins-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-tools-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-python-admin-extensions-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-python-plugins-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-rpm-admin-extensions-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-rpm-plugins-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-selinux-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-server-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
401c401
< python2-crypto-2.6.1-15.el7.x86_64
---
> python2-crypto-2.6.1-13.el7.x86_64
406c406
< python2-jmespath-0.9.0-3.el7.noarch
---
> python2-jmespath-0.9.0-1.el7.noarch
427c427
< python-crane-3.3.0-1.el7.noarch
---
> python-crane-3.3.0-0.1.alpha.201902110520git579a2a1.el7.noarch
443c443
< python-httplib2-0.9.2-1.el7.noarch
---
> python-httplib2-0.9.2-0.1.el7.noarch
469,480c469,480
< python-pulp-bindings-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-client-lib-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-deb-common-1.8.0-1.el7.noarch
< python-pulp-docker-common-3.2.2-1.el7.noarch
< python-pulp-oid_validation-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-ostree-common-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-puppet-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-python-common-2.0.3-1.el7.noarch
< python-pulp-repoauth-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-rpm-common-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< python-pulp-streamer-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> python-pulp-bindings-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-client-lib-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-common-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-deb-common-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> python-pulp-docker-common-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> python-pulp-oid_validation-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-ostree-common-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> python-pulp-puppet-common-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> python-pulp-python-common-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> python-pulp-repoauth-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-rpm-common-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> python-pulp-streamer-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
559c559
< sshpass-1.06-2.el7.x86_64
---
> sshpass-1.06-1.el7.x86_64

#2 Updated by dalley 12 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

I reproduced this on CentOS, but could not reproduce it on Fedora, as with many other bugs recently, most notably the libmodulemd one. I'm not really sure what's up with that.

#3 Updated by dalley 12 months ago

I'm fairly certain this is a libsolv problem. 0.7.2 (Fedora) works, 0.6.34 does not, with the same Pulp code.

Discussion with Igor:

<dalley> so I think it might actually be an already resolved libsolv issue -- it works perfectly on Fedora 29 w/ v0.7.2, but segfaults on CentOS with 0.6.34-2
<ignatenkobrain> forget about 0.6.x =)
<dalley> it's what CentOS provides :) is it something that could be safely upgraded if I pinged the maintainer, or should we just vendor it in our own repo?
<ignatenkobrain> 0.7 bumps soname because it removed few functions which were part of ABI
<ignatenkobrain> but nobody used them for last 5 years
<ignatenkobrain> so for now you can just develop code on f29 and later just decide how to proceed with EL

#4 Updated by CodeHeeler 12 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 48

#5 Updated by CodeHeeler 12 months ago

  • Blocks Release 2.19.z added

#6 Updated by pcreech 12 months ago

After working through various builds of libsolv, I'm suggesting we update the libsolv dependency to the 0.7.x version.

I'm usure why the segfault happens, but in various builds < 0.7.0, it occured.

#7 Updated by dalley 12 months ago

  • Assignee changed from dalley to pcreech

#8 Updated by rchan 11 months ago

  • Sprint changed from Sprint 48 to Sprint 49

#9 Updated by bherring 11 months ago

  • Blocks Test #4364: Implement modularity content dependency solving added

#10 Updated by bherring 11 months ago

  • Status changed from ASSIGNED to MODIFIED

Update

pcreech built libsolv. Tested with the version in 2-master with #4405 tests commenting out the skip.

Tests passed without issue.

Moving to `MODIFIED`.

#11 Updated by ttereshc 11 months ago

  • Platform Release set to 2.19.0

#12 Updated by ttereshc 11 months ago

  • Sprint/Milestone set to 2.19.0

#13 Updated by ttereshc 10 months ago

  • Status changed from MODIFIED to ON_QA

#14 Updated by ttereshc 10 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter 10 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF