Project

Profile

Help

Issue #3036

closed

Pulp streamer dumps core on F26

Added by Ichimonji10 over 6 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.14.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 26
Quarter:

Description

One can install Pulp 2.14 on F26 and successfully use most features. Unfortunately, the pulp_streamer service is broken. One way to reproduce this issue is to run any of the test cases that make use of the streamer. For example:

$ python -m unittest pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase
E
======================================================================                                                                                                                                                                         
ERROR: setUpClass (pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase)                                                                                                                                                      
----------------------------------------------------------------------                                                                                                                                                                         
Traceback (most recent call last):                                                                                     
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/test_download_policies.py", line 101, in setUpClass                                                                                                                      
    cls.rpm = get_unit(cls.cfg, cls.repo['distributors'][0], RPM)                                                                                                                                                                              
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/utils.py", line 377, in get_unit                                                                                                                                         
    return api.Client(cfg).get(path)                                                                                                                                                                                                           
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 285, in get                                                                                                                                                                 
    return self.request('GET', url, **kwargs)                                                                                                                                                                                                  
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 342, in request                                                                                                                                                             
    requests.request(method, **request_kwargs),                                                                                                                                                                                                
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 119, in safe_handler                                                                                                                                                        
    response.raise_for_status()                                                                                                                                                                                                                
  File "/home/ichimonji10/.venvs/pulp-smash/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status                                                                                                                     
    raise HTTPError(http_error_msg, response=self)                                                                                                                                                                                             
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://fedora-26-pulp-2-14:443/streamer/var/lib/pulp/content/units/rpm/c6/c0929d9f5fdb7afb53adb48ebf8b3efd69b5c76f7728b430419e9b62f49544/bear-4.1-1.noarch.rpm
?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTkyLjE2OC4xMDAuMSJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9jNi9jMDkyOWQ5ZjVmZGI3YWZiNTNhZGI0OGViZjhiM2VmZDY5YjVjNzZmNzcyOGI0MzA0MTllOWI2MmY0OTU0NC9iZWFyLTQ
uMS0xLm5vYXJjaC5ycG0iLCAiZXhwaXJhdGlvbiI6IDE1MDY0NTk3MDJ9;signature=Ww3pSRqvX-0Z4j40ZgYACFojkB8wgn_X09u5aMWiSjHeSQaKHh8sq7P3u9B88zZft18Yl2AQQ4SpnNcnvasgcBYWV7ZJK-2Nzf1hXYaPf89IXREXYWOwPUBBcTLDqgE6WQmV3iOCb32Ski4hnslGuRCD60Qk8Jr8m33Ouq7Gf64
gWHpVab9LNs84MhH_1acq2n-HkT2uEXIr70PifaMu2F0ftgKIqZ32WkrhuFGwufYysZNXK242kTve33a-VvJBQKyrbAOFkkDOx6HxyriKNyk28ysz_VOzVJszSmNKlRz2aLrtLmCrp-uleDTUTecwM8lGQ2MNlCmaqtAwF_y8lA%3D%3D                                                              

----------------------------------------------------------------------                                                                                                                                                                         
Ran 0 tests in 48.936s                                                                                                                                                                                                                         

FAILED (errors=1)

That looks nasty. Let's take a look at logs:

[root@fedora-26-pulp-2-14 ~]# journalctl -p err..crit
(snip!)
Sep 26 17:00:05 fedora-26-pulp-2-14 systemd-coredump[1580]: Process 30421 (pulp_streamer) of user 48 dumped core.
(snip!)

That's no fun. I've attached the full output of this journalctl command. It's 518 lines long. Take a look. The end effect is that pulp_streamer.service is dead:

[root@fedora-26-pulp-2-14 ~]# systemctl --state failed
  UNIT                  LOAD   ACTIVE SUB    DESCRIPTION                           
● pulp_streamer.service loaded failed failed The Pulp lazy content loading streamer

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Files

journalctl.txt (63.6 KB) journalctl.txt Ichimonji10, 09/26/2017 11:17 PM
Actions #1

Updated by Ichimonji10 over 6 years ago

  • Description updated (diff)
Actions #2

Updated by amacdona@redhat.com over 6 years ago

  • Sprint/Milestone set to 45
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes
Actions #3

Updated by dalley over 6 years ago

  • Status changed from NEW to ASSIGNED
Actions #4

Updated by Ichimonji10 over 6 years ago

dalley I just reproduced the issue with an installation of the 2.14 beta by applying the following diff to Pulp Smash and executing `python -m unittest pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase`:

diff --git a/pulp_smash/tests/rpm/api_v2/test_download_policies.py b/pulp_smash/tests/rpm/api_v2/test_download_policies.py
index 3fa61d1..6ebd7bf 100644                              
--- a/pulp_smash/tests/rpm/api_v2/test_download_policies.py                                                            
+++ b/pulp_smash/tests/rpm/api_v2/test_download_policies.py                                                            
@@ -37,9 +37,6 @@ def setUpModule():  # pylint:disable=invalid-name                                                    
     cfg = config.get_config()                             
     if cfg.version < Version('2.8'):                      
         raise unittest.SkipTest('This module requires Pulp 2.8 or greater.')                                          
-    if (utils.os_is_f26(cfg) and                          
-            selectors.bug_is_untestable(3036, cfg.version)):                                                          
-        raise unittest.SkipTest('https://pulp.plan.io/issues/3036')                                                   
     if check_issue_2798(cfg):                             
         raise unittest.SkipTest('https://pulp.plan.io/issues/2798')                                                   
     if check_issue_2387(cfg):
Actions #5

Updated by ttereshc over 6 years ago

  • Assignee set to dalley
Actions #6

Updated by Ichimonji10 over 6 years ago

python -m unittest pulp_smash.tests.rpm.api_v2.test_download_policies.BackgroundTestCase succeeds on F26 when SELinux is disabled.

Actions #7

Updated by Ichimonji10 over 6 years ago

I executed the following on several hosts, followed by the problematic tests:

setenforce 0
echo > /var/log/audit/audit.log
semodule -R

audit2allow -al produces no output. Neither does audit2allow -Ral. /var/log/audit/audit.log contains little of interest:

[root@fedora-26-pulp-2-14-beta ~]# grep -i avc /var/log/audit/audit.log
type=USER_AVC msg=audit(1507222031.104:2965): pid=660 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  received policyload notice (seqno=11)  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1507222037.986:2997): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  received policyload notice (seqno=11)  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[root@fedora-26-pulp-2-14-beta ~]# grep -i deni /var/log/audit/audit.log

The same output is seen on Fedora 24 and 25, and nearly the same output is seen on RHEL 7. The system logs are more interesting:

[root@fedora-26-pulp-2-14-beta ~]# journalctl | grep -i deni
Oct 05 10:57:18 fedora-26-pulp-2-14-beta audit[30388]: AVC avc:  denied  { read } for  pid=30388 comm="pulp_streamer" name="cpu" dev="sysfs" ino=33 scontext=system_u:system_r:streamer_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=dir permissive=0
Oct 05 12:40:42 fedora-26-pulp-2-14-beta audit[30388]: AVC avc:  denied  { execmem } for  pid=30388 comm="pulp_streamer" scontext=system_u:system_r:streamer_t:s0 tcontext=system_u:system_r:streamer_t:s0 tclass=process permissive=1

This command produces no output on F24, F25 and RHEL 7.

Added by dalley over 6 years ago

Revision d58bd568 | View on GitHub

Fixed selinux permissions for pulp-streamer on F26

closes #3036 https://pulp.plan.io/issues/3036

Actions #8

Updated by dalley over 6 years ago

  • Status changed from ASSIGNED to POST
Actions #9

Updated by dalley over 6 years ago

  • Status changed from POST to MODIFIED
Actions #10

Updated by pcreech over 6 years ago

  • Platform Release set to 2.14.2
Actions #11

Updated by pcreech over 6 years ago

  • Status changed from MODIFIED to 5
Actions #12

Updated by pthomas@redhat.com over 6 years ago

Automated tests for streamer passed on F26.

Actions #14

Updated by pcreech over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #15

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 26
Actions #16

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (45)
Actions #17

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF