Project

Profile

Help

Issue #4405

Pulp 2.19master nightly ci regression at test_modularity.py

Added by bherring 9 months ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.19.0
Blocks Release:
2.19.z
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 49

Description

Failure Scenario

On nightly ci, test_modularity.py runs on 2.19master and fails in such a way that the box is no longer responsive to API testing.

Note this test works and was re-tested after the above error on 2.19 without issue on 2.18.1b1.

Information:

Failure Seen

  • Test Harness locally (RHEL76 base + Pulp 2 installed, non-FIPS) fails and dies at test_modularity
  • Jenkins also fails at the same location killing the test harness

journalctl output at time of failure

Feb 11 01:42:00 host-172-16-46-47.openstacklocal kernel: celery[12053]: segfault at 0 ip 00007f9b1e4a8b99 sp 00007ffe64b687e0 error 4 in libsolv.so.0[7f9b1e43f000+89000]
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:00 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork.
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)   "MongoClient opened before fork. Create MongoClient " 
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: py.warnings:WARNING: [9cbb7189] (30621-64032)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: pulp.server.async.tasks:INFO: [9cbb7189] Task failed : [0349bb3b-ebb7-4585-810d-8b46ed94cb8f] : Worker terminated abnormally while processing task 03
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Task pulp.server.async.tasks._release_resource[9cbb7189-331f-42f5-b957-0d05211d5c62] raised unexpect
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) Traceback (most recent call last):
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     R = retval = fun(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return super(PulpTask, self).__call__(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     return self.run(*args, **kwargs)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 296, in _release_resource
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     new_task.on_failure(exception, task_id, (), {}, MyEinfo)
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 602, in on_failure
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     if not self.request.called_directly:
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 978, in _get_request
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032)     req = self.request_stack.top
Feb 11 01:42:01 host-172-16-46-47.openstacklocal pulp[30621]: celery.app.trace:ERROR: [9cbb7189] (30621-64032) AttributeError: 'NoneType' object has no attribute 'top'
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) Traceback (most recent call last):
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032)     human_status(exitcode)),
Feb 11 01:42:11 host-172-16-46-47.openstacklocal pulp[11083]: celery.worker.request:ERROR: (11083-64032) WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

Related issues

Blocks RPM Support - Test #4364: Implement modularity content dependency solving CLOSED - COMPLETE Actions

Associated revisions

Revision 92118dd8 View on GitHub
Added by bherring 9 months ago

Skipping modular tests due to #4405

Issue #4405 is a regression due to libsolv in RHEL releases
affecting all modular tests and hanging Jenkins CI.

Skipping tests until the issue is resolved.

refs #4405

History

#1 Updated by bherring 9 months ago

Adding 2.18.1b1 vs 2.19-master diff

[root@rhel76-2181b1 ~]# diff 218b1.rpms.list  219master.rpms.list 
230,231c230,231
< libtomcrypt-1.17-26.el7.x86_64
< libtommath-0.42.0-6.el7.x86_64
---
> libtomcrypt-1.17-25.el7.x86_64
> libtommath-0.42.0-5.el7.x86_64
372,387c372,387
< pulp-admin-client-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-deb-admin-extensions-1.8.0-1.el7.noarch
< pulp-deb-plugins-1.8.0-1.el7.noarch
< pulp-docker-admin-extensions-3.2.2-1.el7.noarch
< pulp-docker-plugins-3.2.2-1.el7.noarch
< pulp-ostree-admin-extensions-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-ostree-plugins-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-admin-extensions-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-plugins-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-puppet-tools-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-python-admin-extensions-2.0.3-1.el7.noarch
< pulp-python-plugins-2.0.3-1.el7.noarch
< pulp-rpm-admin-extensions-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-rpm-plugins-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< pulp-selinux-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< pulp-server-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> pulp-admin-client-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-deb-admin-extensions-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-deb-plugins-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> pulp-docker-admin-extensions-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-docker-plugins-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-ostree-plugins-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> pulp-puppet-admin-extensions-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-plugins-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-puppet-tools-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> pulp-python-admin-extensions-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-python-plugins-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> pulp-rpm-admin-extensions-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-rpm-plugins-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> pulp-selinux-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> pulp-server-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
401c401
< python2-crypto-2.6.1-15.el7.x86_64
---
> python2-crypto-2.6.1-13.el7.x86_64
406c406
< python2-jmespath-0.9.0-3.el7.noarch
---
> python2-jmespath-0.9.0-1.el7.noarch
427c427
< python-crane-3.3.0-1.el7.noarch
---
> python-crane-3.3.0-0.1.alpha.201902110520git579a2a1.el7.noarch
443c443
< python-httplib2-0.9.2-1.el7.noarch
---
> python-httplib2-0.9.2-0.1.el7.noarch
469,480c469,480
< python-pulp-bindings-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-client-lib-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-deb-common-1.8.0-1.el7.noarch
< python-pulp-docker-common-3.2.2-1.el7.noarch
< python-pulp-oid_validation-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-ostree-common-1.4.0-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-puppet-common-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-python-common-2.0.3-1.el7.noarch
< python-pulp-repoauth-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
< python-pulp-rpm-common-2.18.1-0.2.beta.git.105.fc6457c.el7.noarch
< python-pulp-streamer-2.18.1-0.1.beta.git.104.2e73e54.el7.noarch
---
> python-pulp-bindings-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-client-lib-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-common-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-deb-common-1.9.0-0.1.alpha.201902110523gite0d9027.el7.noarch
> python-pulp-docker-common-3.4.0-0.1.alpha.201902110501git9d72e94.el7.noarch
> python-pulp-oid_validation-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-ostree-common-1.4.0-0.1.alpha.201902110504git3377449.el7.noarch
> python-pulp-puppet-common-2.19.0-0.1.alpha.201902110506gitc8c9a9a.el7.noarch
> python-pulp-python-common-2.1.0-0.1.alpha.201902110500git5e2aa35.el7.noarch
> python-pulp-repoauth-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
> python-pulp-rpm-common-2.19.0-0.1.alpha.201902110510gitc5f2780.el7.noarch
> python-pulp-streamer-2.19.0-0.1.alpha.201902110515git47da396.el7.noarch
559c559
< sshpass-1.06-2.el7.x86_64
---
> sshpass-1.06-1.el7.x86_64

#2 Updated by dalley 9 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

I reproduced this on CentOS, but could not reproduce it on Fedora, as with many other bugs recently, most notably the libmodulemd one. I'm not really sure what's up with that.

#3 Updated by dalley 9 months ago

I'm fairly certain this is a libsolv problem. 0.7.2 (Fedora) works, 0.6.34 does not, with the same Pulp code.

Discussion with Igor:

<dalley> so I think it might actually be an already resolved libsolv issue -- it works perfectly on Fedora 29 w/ v0.7.2, but segfaults on CentOS with 0.6.34-2
<ignatenkobrain> forget about 0.6.x =)
<dalley> it's what CentOS provides :) is it something that could be safely upgraded if I pinged the maintainer, or should we just vendor it in our own repo?
<ignatenkobrain> 0.7 bumps soname because it removed few functions which were part of ABI
<ignatenkobrain> but nobody used them for last 5 years
<ignatenkobrain> so for now you can just develop code on f29 and later just decide how to proceed with EL

#4 Updated by CodeHeeler 9 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 48

#5 Updated by CodeHeeler 9 months ago

  • Blocks Release 2.19.z added

#6 Updated by pcreech 9 months ago

After working through various builds of libsolv, I'm suggesting we update the libsolv dependency to the 0.7.x version.

I'm usure why the segfault happens, but in various builds < 0.7.0, it occured.

#7 Updated by dalley 9 months ago

  • Assignee changed from dalley to pcreech

#8 Updated by rchan 9 months ago

  • Sprint changed from Sprint 48 to Sprint 49

#9 Updated by bherring 9 months ago

  • Blocks Test #4364: Implement modularity content dependency solving added

#10 Updated by bherring 9 months ago

  • Status changed from ASSIGNED to MODIFIED

Update

pcreech built libsolv. Tested with the version in 2-master with #4405 tests commenting out the skip.

Tests passed without issue.

Moving to `MODIFIED`.

#11 Updated by ttereshc 8 months ago

  • Platform Release set to 2.19.0

#12 Updated by ttereshc 8 months ago

  • Sprint/Milestone set to 2.19.0

#13 Updated by ttereshc 8 months ago

  • Status changed from MODIFIED to ON_QA

#14 Updated by ttereshc 8 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter 7 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF