Pulp

Let's say one executes the following script:

pulp-admin rpm repo create --repo-id foo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm/
pulp-admin rpm repo sync run --repo-id foo
wget --no-check-certificate https://localhost/pulp/repos/pulp/pulp/fixtures/rpm/bear-4.1-1.noarch.rpm
pulp-admin rpm repo delete --repo-id foo

This should work, but it doesn't under the currently nightly builds of Pulp 2.10 and 2.11. The wget step fails with an HTTP 404. Why does it fail? Because of an SELinux denial. If SELinux is disabled, the script above succeeds.

It's not just the RPM plugin that suffers from SELinux denials. Here's some journal entries from a failed RPM repository publish:

Sep 22 11:39:14 example.com pulp[8792]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[786fdf54-f9f8-45ad-b974-8151010cb408]
Sep 22 11:39:14 example.com pulp[8730]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[80db8bf1-7f2f-4625-b673-30893b06bb1b]
Sep 22 11:39:14 example.com pulp[8730]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[22a530c5-a514-437c-a554-999b049b1e13]
Sep 22 11:39:14 example.com pulp[8792]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[786fdf54-f9f8-45ad-b974-8151010cb408] succeeded in 0.0355534609998s: None
Sep 22 11:39:14 example.com pulp[8730]: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[80db8bf1-7f2f-4625-b673-30893b06bb1b] succeeded in 0.188737557s: {'exception': None, 'repo_id': u'15421972-73ad-45d8-a908-1f583ee01ea3', 'traceback': None, 'started': '2016-09-22T15:39:14Z',...
Sep 22 11:39:14 example.com pulp[8730]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[22a530c5-a514-437c-a554-999b049b1e13] succeeded in 0.0058686479997s: None
Sep 22 11:39:24 example.com audit[8990]: AVC avc:  denied  { getattr } for  pid=8990 comm="httpd" path="/var/lib/pulp/published/yum/master/yum_distributor/77e0fa17-6fd4-4631-ae43-500355556f68/1474558752.03/bear-4.1-1.noarch.rpm" dev="dm-0" ino=658416 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=lnk_file permissive=0
Sep 22 11:39:24 example.com audit[8990]: AVC avc:  denied  { getattr } for  pid=8990 comm="httpd" path="/var/lib/pulp/published/yum/master/yum_distributor/77e0fa17-6fd4-4631-ae43-500355556f68/1474558752.03/bear-4.1-1.noarch.rpm" dev="dm-0" ino=658416 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=lnk_file permissive=0
Sep 22 11:39:24 example.com pulp[8857]: django.request:WARNING: Not Found: /var/www/pub/yum/https/repos/ddb5529c-85de-43e1-8dcc-fa2920f5d23a/bear-4.1-1.noarch.rpm
Sep 22 11:39:27 example.com audit[9039]: AVC avc:  denied  { getattr } for  pid=9039 comm="httpd" path="/var/lib/pulp/published/yum/master/yum_distributor/15421972-73ad-45d8-a908-1f583ee01ea3/1474558754.52/bear-4.1-1.noarch.rpm" dev="dm-0" ino=658438 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=lnk_file permissive=0
Sep 22 11:39:27 example.com audit[9039]: AVC avc:  denied  { getattr } for  pid=9039 comm="httpd" path="/var/lib/pulp/published/yum/master/yum_distributor/15421972-73ad-45d8-a908-1f583ee01ea3/1474558754.52/bear-4.1-1.noarch.rpm" dev="dm-0" ino=658438 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=lnk_file permissive=0
Sep 22 11:39:27 example.com pulp[8856]: django.request:WARNING: Not Found: /var/www/pub/yum/https/repos/b14dbebb-3cf6-4974-ac6c-2cd6f6198b61/bear-4.1-1.noarch.rpm
Sep 22 11:39:29 example.com pulp[8854]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS

And here's some journal entries from a failed Docker image publish:

Sep 22 13:27:02 example.com pulp[12819]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[9e5329e7-b9ee-4e4a-a11f-eb443b8e9019]
Sep 22 13:27:02 example.com pulp[12819]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[cb9f6aa3-47d3-4701-a845-5da21e1ea713]
Sep 22 13:27:02 example.com pulp[12684]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[5f56d2e7-7c04-4d57-972b-f6a5bc2e3c8d] succeeded in 0.0367998849997s: None
Sep 22 13:27:03 example.com pulp[12819]: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[9e5329e7-b9ee-4e4a-a11f-eb443b8e9019] succeeded in 0.712945763002s: {'exception': None, 'repo_id': u'65e8a280-d78c-4d8e-9454-4f20e5672870', 'traceback': None, 'started': '2016-09-22T17:27:02Z',...
Sep 22 13:27:03 example.com pulp[12819]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[cb9f6aa3-47d3-4701-a845-5da21e1ea713] succeeded in 0.00820000000022s: None
Sep 22 13:27:05 example.com sudo[13755]:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /var/lib/pulp/published/docker/v2/app/65e8a280-d78c-4d8e-9454-4f20e5672870.json
Sep 22 13:27:05 example.com audit[13755]: USER_CMD pid=13755 uid=0 auid=0 ses=5 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd="/root" cmd=636174202F7661722F6C69622F70756C702F7075626C69736865642F646F636B65722F76322F6170702F36356538613238302D643738632D346438652D393435342D3466323065353637323837302E6A736F6E terminal=? res=success'
Sep 22 13:27:05 example.com audit[12728]: AVC avc:  denied  { getattr } for  pid=12728 comm="httpd" path="/var/lib/pulp/published/docker/v2/master/65e8a280-d78c-4d8e-9454-4f20e5672870/1474565222.52/tags/list" dev="dm-0" ino=1073976 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=file permissive=0
Sep 22 13:27:05 example.com audit[12728]: AVC avc:  denied  { getattr } for  pid=12728 comm="httpd" path="/var/lib/pulp/published/docker/v2/master/65e8a280-d78c-4d8e-9454-4f20e5672870/1474565222.52/tags/list" dev="dm-0" ino=1073976 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=file permissive=0

Both of the failed publishes generated AVC avc: denied journal entries.

Here's a more full-fledged script for determining exactly what's wrong:

#!/usr/bin/env bash
#
# Demonstrate the issue with Pulp and get information about it. All statements
# with `|| true` prepended are known to return non-zero exit codes.
#
set -euo pipefail

# Configure SELinux
setenforce 0
echo > /var/log/audit/audit.log
semodule -R

# Execute problematic actions
pulp-admin rpm repo create --repo-id foo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm/
pulp-admin rpm repo sync run --repo-id foo
wget --no-check-certificate https://localhost/pulp/repos/pulp/pulp/fixtures/rpm/bear-4.1-1.noarch.rpm
pulp-admin rpm repo delete --repo-id foo

# Get info
set -x
audit2allow -al
audit2allow -Ral || true
cat /var/log/audit/audit.log

In this script, SELinux is disabled, and as a result, the publish and subsequent wget succeed. Here's the last few lines of output:

+ audit2allow -al

#============= httpd_t ==============
allow httpd_t pulp_var_cache_t:lnk_file { read getattr };
+ audit2allow -Ral
could not open interface info [/var/lib/sepolgen/interface_info]
+ true
+ cat /var/log/audit/audit.log

type=MAC_POLICY_LOAD msg=audit(1474569379.871:2460): policy loaded auid=0 ses=7
type=AVC msg=audit(1474569390.690:2461): avc:  denied  { getattr } for  pid=23325 comm="httpd" path="/var/lib/pulp/published/yum/master/yum_distributor/foo/1474569388.72/bear-4.1-1.noarch.rpm" dev="dm-0" ino=658518 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=lnk_file permissive=1
type=AVC msg=audit(1474569390.690:2462): avc:  denied  { read } for  pid=23325 comm="httpd" name="bear-4.1-1.noarch.rpm" dev="dm-0" ino=658518 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=lnk_file permissive=1

For constrast, when the same script is run on a days-old Pulp system that doesn't suffer from this issue, here's what the last few lines look like:

+ audit2allow -al

+ audit2allow -Ral
could not open interface info [/var/lib/sepolgen/interface_info]
+ true
+ cat /var/log/audit/audit.log

type=USER_AVC msg=audit(1474569380.233:7714): pid=670 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  received policyload notice (seqno=11)  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
type=MAC_POLICY_LOAD msg=audit(1474569380.238:7715): policy loaded auid=0 ses=105

Here's the packages on the system I used to reproduce this bug. (Jenkins shows that this issue applied to all Pulp 2.10 and 2.11 nightly builds across all OS distributions.)

$ grep PRETTY /etc/os-release && rpm -qa | sort | grep -i pulp
PRETTY_NAME="Fedora 23 (Twenty Three)"
pulp-admin-client-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
pulp-docker-admin-extensions-2.2.0-0.1.alpha.git.201.d0860fd.fc23.noarch
pulp-docker-plugins-2.2.0-0.1.alpha.git.201.d0860fd.fc23.noarch
pulp-ostree-admin-extensions-1.2.0-0.1.alpha.git.104.42d1c09.fc23.noarch
pulp-ostree-plugins-1.2.0-0.1.alpha.git.104.42d1c09.fc23.noarch
pulp-puppet-admin-extensions-2.11.0-0.1.alpha.git.187.e97c179.fc23.noarch
pulp-puppet-plugins-2.11.0-0.1.alpha.git.187.e97c179.fc23.noarch
pulp-python-admin-extensions-1.2.0-0.1.alpha.git.108.182206a.fc23.noarch
pulp-python-plugins-1.2.0-0.1.alpha.git.108.182206a.fc23.noarch
pulp-rpm-admin-extensions-2.11.0-0.1.alpha.git.542.ceaaaca.fc23.noarch
pulp-rpm-plugins-2.11.0-0.1.alpha.git.542.ceaaaca.fc23.noarch
pulp-selinux-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
pulp-server-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
python-kombu-3.0.33-6.pulp.fc23.noarch
python-pulp-bindings-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
python-pulp-client-lib-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
python-pulp-common-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
python-pulp-docker-common-2.2.0-0.1.alpha.git.201.d0860fd.fc23.noarch
python-pulp-oid_validation-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
python-pulp-ostree-common-1.2.0-0.1.alpha.git.104.42d1c09.fc23.noarch
python-pulp-puppet-common-2.11.0-0.1.alpha.git.187.e97c179.fc23.noarch
python-pulp-python-common-1.2.0-0.1.alpha.git.108.182206a.fc23.noarch
python-pulp-repoauth-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch
python-pulp-rpm-common-2.11.0-0.1.alpha.git.542.ceaaaca.fc23.noarch
python-pulp-streamer-2.11.0-0.1.alpha.git.701.0c0ee81.fc23.noarch