Issue #3036
closedPulp streamer dumps core on F26
Description
One can install Pulp 2.14 on F26 and successfully use most features. Unfortunately, the pulp_streamer service is broken. One way to reproduce this issue is to run any of the test cases that make use of the streamer. For example:
$ python -m unittest pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase
E
======================================================================
ERROR: setUpClass (pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/test_download_policies.py", line 101, in setUpClass
cls.rpm = get_unit(cls.cfg, cls.repo['distributors'][0], RPM)
File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/utils.py", line 377, in get_unit
return api.Client(cfg).get(path)
File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 285, in get
return self.request('GET', url, **kwargs)
File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 342, in request
requests.request(method, **request_kwargs),
File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 119, in safe_handler
response.raise_for_status()
File "/home/ichimonji10/.venvs/pulp-smash/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://fedora-26-pulp-2-14:443/streamer/var/lib/pulp/content/units/rpm/c6/c0929d9f5fdb7afb53adb48ebf8b3efd69b5c76f7728b430419e9b62f49544/bear-4.1-1.noarch.rpm
?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTkyLjE2OC4xMDAuMSJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9jNi9jMDkyOWQ5ZjVmZGI3YWZiNTNhZGI0OGViZjhiM2VmZDY5YjVjNzZmNzcyOGI0MzA0MTllOWI2MmY0OTU0NC9iZWFyLTQ
uMS0xLm5vYXJjaC5ycG0iLCAiZXhwaXJhdGlvbiI6IDE1MDY0NTk3MDJ9;signature=Ww3pSRqvX-0Z4j40ZgYACFojkB8wgn_X09u5aMWiSjHeSQaKHh8sq7P3u9B88zZft18Yl2AQQ4SpnNcnvasgcBYWV7ZJK-2Nzf1hXYaPf89IXREXYWOwPUBBcTLDqgE6WQmV3iOCb32Ski4hnslGuRCD60Qk8Jr8m33Ouq7Gf64
gWHpVab9LNs84MhH_1acq2n-HkT2uEXIr70PifaMu2F0ftgKIqZ32WkrhuFGwufYysZNXK242kTve33a-VvJBQKyrbAOFkkDOx6HxyriKNyk28ysz_VOzVJszSmNKlRz2aLrtLmCrp-uleDTUTecwM8lGQ2MNlCmaqtAwF_y8lA%3D%3D
----------------------------------------------------------------------
Ran 0 tests in 48.936s
FAILED (errors=1)
That looks nasty. Let's take a look at logs:
[root@fedora-26-pulp-2-14 ~]# journalctl -p err..crit
(snip!)
Sep 26 17:00:05 fedora-26-pulp-2-14 systemd-coredump[1580]: Process 30421 (pulp_streamer) of user 48 dumped core.
(snip!)
That's no fun. I've attached the full output of this journalctl
command. It's 518 lines long. Take a look. The end effect is that pulp_streamer.service
is dead:
[root@fedora-26-pulp-2-14 ~]# systemctl --state failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● pulp_streamer.service loaded failed failed The Pulp lazy content loading streamer
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Files
Updated by amacdona@redhat.com over 7 years ago
- Sprint/Milestone set to 45
- Severity changed from 2. Medium to 3. High
- Triaged changed from No to Yes
Updated by Ichimonji10 over 7 years ago
dalley I just reproduced the issue with an installation of the 2.14 beta by applying the following diff to Pulp Smash and executing `python -m unittest pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase`:
diff --git a/pulp_smash/tests/rpm/api_v2/test_download_policies.py b/pulp_smash/tests/rpm/api_v2/test_download_policies.py
index 3fa61d1..6ebd7bf 100644
--- a/pulp_smash/tests/rpm/api_v2/test_download_policies.py
+++ b/pulp_smash/tests/rpm/api_v2/test_download_policies.py
@@ -37,9 +37,6 @@ def setUpModule(): # pylint:disable=invalid-name
cfg = config.get_config()
if cfg.version < Version('2.8'):
raise unittest.SkipTest('This module requires Pulp 2.8 or greater.')
- if (utils.os_is_f26(cfg) and
- selectors.bug_is_untestable(3036, cfg.version)):
- raise unittest.SkipTest('https://pulp.plan.io/issues/3036')
if check_issue_2798(cfg):
raise unittest.SkipTest('https://pulp.plan.io/issues/2798')
if check_issue_2387(cfg):
Updated by Ichimonji10 over 7 years ago
python -m unittest pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase
succeeds on F26 when SELinux is disabled.
Updated by Ichimonji10 over 7 years ago
I executed the following on several hosts, followed by the problematic tests:
setenforce 0
echo > /var/log/audit/audit.log
semodule -R
audit2allow -al
produces no output. Neither does audit2allow -Ral
. /var/log/audit/audit.log
contains little of interest:
[root@fedora-26-pulp-2-14-beta ~]# grep -i avc /var/log/audit/audit.log
type=USER_AVC msg=audit(1507222031.104:2965): pid=660 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc: received policyload notice (seqno=11) exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1507222037.986:2997): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: received policyload notice (seqno=11) exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[root@fedora-26-pulp-2-14-beta ~]# grep -i deni /var/log/audit/audit.log
The same output is seen on Fedora 24 and 25, and nearly the same output is seen on RHEL 7. The system logs are more interesting:
[root@fedora-26-pulp-2-14-beta ~]# journalctl | grep -i deni
Oct 05 10:57:18 fedora-26-pulp-2-14-beta audit[30388]: AVC avc: denied { read } for pid=30388 comm="pulp_streamer" name="cpu" dev="sysfs" ino=33 scontext=system_u:system_r:streamer_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=dir permissive=0
Oct 05 12:40:42 fedora-26-pulp-2-14-beta audit[30388]: AVC avc: denied { execmem } for pid=30388 comm="pulp_streamer" scontext=system_u:system_r:streamer_t:s0 tcontext=system_u:system_r:streamer_t:s0 tclass=process permissive=1
This command produces no output on F24, F25 and RHEL 7.
Added by dalley over 7 years ago
Updated by dalley over 7 years ago
- Status changed from ASSIGNED to POST
Updated by dalley over 7 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp|d58bd568a92326142e342c2af80b66e61f42eb8c.
Updated by pthomas@redhat.com over 7 years ago
Automated tests for streamer passed on F26.
Updated by pcreech over 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Fixed selinux permissions for pulp-streamer on F26
closes #3036 https://pulp.plan.io/issues/3036