Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2287

closed

Cannot get docker v2 repo tags list

Added by Ichimonji10 over 7 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Assignee:
dkliban@redhat.com
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.10.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

TL;DR version: HTTP GET requests to /pulp/docker/v2/foo/tags/list/ always fail with an HTTP 403 (forbidden) due to SELinux issues.

The longer version follows.

Let's say one executes the following script:

# Execute problematic actions
pulp-admin docker repo create --repo-id foo \
    --feed https://registry-1.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id foo
curl --insecure --user admin:admin \
    https://localhost/pulp/docker/v2/foo/tags/list/
pulp-admin docker repo delete --repo-id foo

This should work, but it doesn't under the current nightly builds of Pulp 2.10 and 2.11. The curl step fails with an HTTP 403 (forbidden). Here's what's printed:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /pulp/docker/v2/foo/tags/list/ was not found on this server.</p>
</body></html>

Why does this happen? That's unclear, but a likely cause is an SELinux denial. If SELinux is disabled, the script above succeeds. Here's a more full-fledged script that succeeds:

#!/usr/bin/env bash
set -euo pipefail

# Configure SELinux
setenforce 0
echo > /var/log/audit/audit.log
semodule -R

# Execute problematic actions
pulp-admin docker repo create --repo-id foo \
    --feed https://registry-1.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id foo
curl --insecure --user admin:admin \
    https://localhost/pulp/docker/v2/foo/tags/list/
pulp-admin docker repo delete --repo-id foo

# Get info
set -x
audit2allow -al
audit2allow -Ral || true
cat /var/log/audit/audit.log

Here's the last few lines of output:

+ audit2allow -al

#============= httpd_t ==============

#!!!! The file '/var/lib/pulp/published/docker/v2/master/foo/1475082477.3/tags/list' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /var/lib/pulp/published/docker/v2/master/foo/1475082477.3/tags/list
allow httpd_t pulp_var_cache_t:file getattr;
+ audit2allow -Ral
could not open interface info [/var/lib/sepolgen/interface_info]
+ true
+ cat /var/log/audit/audit.log

type=MAC_POLICY_LOAD msg=audit(1475082461.412:12606): policy loaded auid=0 ses=108
type=AVC msg=audit(1475082479.139:12607): avc:  denied  { getattr } for  pid=10392 comm="httpd" path="/var/lib/pulp/published/docker/v2/master/foo/1475082477.3/tags/list" dev="dm-0" ino=1716474 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=file permissive=1

For what it's worth, here's the corresponding log entries from the system, as captured by journalctl --follow:

-- Logs begin at Mon 2016-09-26 09:17:28 EDT. --
Sep 28 13:16:57 pulp.example.com pulp[11859]: pulp.server.agent.direct.services:INFO: Task reply handler, started.
Sep 28 13:16:57 pulp.example.com pulp[11859]: gofer.messaging.adapter.qpid.connection:INFO: open: URL: tcp://localhost|SSL: ca: /etc/pki/qpid/ca/ca.crt|key: None|certificate: /etc/pki/qpid/client/client.pem|host-validation: False
Sep 28 13:16:57 pulp.example.com pulp[11859]: pulp.server.agent.direct.services:INFO: AMQP reply handler started
Sep 28 13:16:57 pulp.example.com pulp[11859]: pulp.server.webservices.application:INFO: *************************************************************
Sep 28 13:16:57 pulp.example.com pulp[11859]: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized
Sep 28 13:16:57 pulp.example.com pulp[11859]: pulp.server.webservices.application:INFO: *************************************************************
Sep 28 13:16:57 pulp.example.com pulp[11859]: gofer.messaging.adapter.qpid.connection:INFO: opened: qpid+tcp://localhost:5672
Sep 28 13:16:57 pulp.example.com pulp[11859]: gofer.messaging.adapter.connect:INFO: connected: qpid+tcp://localhost:5672
Sep 28 13:16:59 pulp.example.com audit[9682]: USER_START pid=9682 uid=0 auid=0 ses=108 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=0 exe="/usr/sbin/sshd" hostname=10.10.53.186 addr=10.10.53.186 terminal=/dev/pts/0 res=success'
Sep 28 13:16:59 pulp.example.com audit[9682]: CRYPTO_KEY_USER pid=9682 uid=0 auid=0 ses=108 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:ef:4c:bc:82:5b:1d:8d:86:96:a5:89:f5:d2:fe:5b:72:bd:4b:2e:4d:35:39:a8:bf:79:85:65:3e:f8:cc:b9:fa direction=? spid=12165 suid=0  exe="/usr/sbin/sshd" hostname=? addr=10.10.53.186 terminal=? res=success'
Sep 28 13:17:10 pulp.example.com kernel: SELinux: 32768 avtab hash slots, 103924 rules.
Sep 28 13:17:10 pulp.example.com kernel: SELinux: 32768 avtab hash slots, 103924 rules.
Sep 28 13:17:10 pulp.example.com kernel: SELinux:  8 users, 14 roles, 4956 types, 308 bools, 1 sens, 1024 cats
Sep 28 13:17:10 pulp.example.com kernel: SELinux:  92 classes, 103924 rules
Sep 28 13:17:10 pulp.example.com kernel: SELinux:  Permission validate_trans in class security not defined in policy.
Sep 28 13:17:10 pulp.example.com kernel: SELinux:  Permission module_load in class system not defined in policy.
Sep 28 13:17:10 pulp.example.com kernel: SELinux:  Class cap_userns not defined in policy.
Sep 28 13:17:10 pulp.example.com kernel: SELinux:  Class cap2_userns not defined in policy.
Sep 28 13:17:10 pulp.example.com kernel: SELinux: the above unknown classes and permissions will be allowed
Sep 28 13:17:11 pulp.example.com dbus[702]: Can't send to audit system: USER_AVC avc:  received policyload notice (seqno=11)
                                                                     exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?
Sep 28 13:17:11 pulp.example.com audit: MAC_POLICY_LOAD policy loaded auid=0 ses=108
Sep 28 13:17:11 pulp.example.com dbus[702]: [system] Reloaded configuration
Sep 28 13:17:11 pulp.example.com pulp[11860]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Sep 28 13:17:11 pulp.example.com pulp[11810]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[74555716-670f-431b-89f8-a4a0ec9e2eda]
Sep 28 13:17:11 pulp.example.com pulp[11956]: py.warnings:WARNING: (11956-10368) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Sep 28 13:17:11 pulp.example.com pulp[11956]: py.warnings:WARNING: (11956-10368)   "MongoClient opened before fork. Create MongoClient "
Sep 28 13:17:11 pulp.example.com pulp[11956]: py.warnings:WARNING: (11956-10368)
Sep 28 13:17:11 pulp.example.com pulp[11956]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Sep 28 13:17:11 pulp.example.com pulp[11752]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[40a8f8a4-a451-4fdb-bca3-2781e702e484]
Sep 28 13:17:11 pulp.example.com pulp[11834]: py.warnings:WARNING: (11834-22336) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Sep 28 13:17:11 pulp.example.com pulp[11834]: py.warnings:WARNING: (11834-22336)   "MongoClient opened before fork. Create MongoClient "
Sep 28 13:17:11 pulp.example.com pulp[11834]: py.warnings:WARNING: (11834-22336)
Sep 28 13:17:11 pulp.example.com pulp[11752]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[c6495e19-f4b7-4456-9b48-a6b71b5ee9ec]
Sep 28 13:17:11 pulp.example.com pulp[11810]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[74555716-670f-431b-89f8-a4a0ec9e2eda] succeeded in 0.0460691690096s: None
Sep 28 13:17:12 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Sep 28 13:17:12 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Sep 28 13:17:12 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Sep 28 13:17:13 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Sep 28 13:17:21 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Sep 28 13:17:21 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Sep 28 13:17:21 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Sep 28 13:17:21 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): registry-1.docker.io
Sep 28 13:17:21 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): registry-1.docker.io
Sep 28 13:17:22 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): registry-1.docker.io
Sep 28 13:17:22 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dseasb33srnrn.cloudfront.net
Sep 28 13:17:24 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): registry-1.docker.io
Sep 28 13:17:24 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): dseasb33srnrn.cloudfront.net
Sep 28 13:17:26 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): registry-1.docker.io
Sep 28 13:17:26 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): registry-1.docker.io
Sep 28 13:17:26 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): dseasb33srnrn.cloudfront.net
Sep 28 13:17:26 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): dseasb33srnrn.cloudfront.net
Sep 28 13:17:26 pulp.example.com pulp[11834]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): dseasb33srnrn.cloudfront.net
Sep 28 13:17:28 pulp.example.com pulp[11834]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Sep 28 13:17:28 pulp.example.com pulp[11810]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[4683a490-5258-4433-ab77-26a5993e56cf]
Sep 28 13:17:28 pulp.example.com pulp[11752]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[40a8f8a4-a451-4fdb-bca3-2781e702e484] succeeded in 16.310618789s: <pulp.server.async.tasks.TaskResult object at 0x7fda4fab9250>
Sep 28 13:17:28 pulp.example.com pulp[11752]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[f9170aca-eb7c-4997-87bf-2a93f952eb25]
Sep 28 13:17:28 pulp.example.com pulp[11752]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[b8fd4289-86f6-4ea7-9410-c56be555377e]
Sep 28 13:17:28 pulp.example.com pulp[11810]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[4683a490-5258-4433-ab77-26a5993e56cf] succeeded in 0.0374169150018s: None
Sep 28 13:17:28 pulp.example.com pulp[11752]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[c6495e19-f4b7-4456-9b48-a6b71b5ee9ec] succeeded in 0.0477471390041s: None
Sep 28 13:17:28 pulp.example.com pulp[11752]: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[f9170aca-eb7c-4997-87bf-2a93f952eb25] succeeded in 0.587830191012s: {'exception': None, 'repo_id': u'foo', 'traceback': None, 'started': '2016-09-28T17:17:28Z', '_ns': u'repo_publish_results',...
Sep 28 13:17:28 pulp.example.com pulp[11752]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b8fd4289-86f6-4ea7-9410-c56be555377e] succeeded in 0.00430754901026s: None
Sep 28 13:17:29 pulp.example.com audit[11871]: AVC avc:  denied  { getattr } for  pid=11871 comm="httpd" path="/var/lib/pulp/published/docker/v2/master/foo/1475083048.37/tags/list" dev="dm-0" ino=1716474 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:pulp_var_cache_t:s0 tclass=file permissive=1

Here's the packages on a system suffering from the issue. The packages are a few days old, but don't worry: the current Beaker systems also suffer from this bug.

ssh $hostname rpm -qa | sort | grep -i pulp
pulp-admin-client-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
pulp-docker-admin-extensions-2.2.0-0.1.alpha.git.201.5462eab.fc23.noarch
pulp-docker-plugins-2.2.0-0.1.alpha.git.201.5462eab.fc23.noarch
pulp-ostree-admin-extensions-1.2.0-0.1.alpha.git.104.8f728f9.fc23.noarch
pulp-ostree-plugins-1.2.0-0.1.alpha.git.104.8f728f9.fc23.noarch
pulp-puppet-admin-extensions-2.11.0-0.1.alpha.git.187.e97c179.fc23.noarch
pulp-puppet-plugins-2.11.0-0.1.alpha.git.187.e97c179.fc23.noarch
pulp-python-admin-extensions-1.2.0-0.1.alpha.git.108.404f437.fc23.noarch
pulp-python-plugins-1.2.0-0.1.alpha.git.108.404f437.fc23.noarch
pulp-rpm-admin-extensions-2.11.0-0.1.alpha.git.543.4255a48.fc23.noarch
pulp-rpm-plugins-2.11.0-0.1.alpha.git.543.4255a48.fc23.noarch
pulp-selinux-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
pulp-server-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
python-kombu-3.0.33-6.pulp.fc23.noarch
python-pulp-bindings-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
python-pulp-client-lib-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
python-pulp-common-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
python-pulp-docker-common-2.2.0-0.1.alpha.git.201.5462eab.fc23.noarch
python-pulp-oid_validation-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
python-pulp-ostree-common-1.2.0-0.1.alpha.git.104.8f728f9.fc23.noarch
python-pulp-puppet-common-2.11.0-0.1.alpha.git.187.e97c179.fc23.noarch
python-pulp-python-common-1.2.0-0.1.alpha.git.108.404f437.fc23.noarch
python-pulp-repoauth-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch
python-pulp-rpm-common-2.11.0-0.1.alpha.git.543.4255a48.fc23.noarch
python-pulp-streamer-2.11.0-0.1.alpha.git.703.7b6e4af.fc23.noarch

Related issues

Related to Pulp - Issue #2277: Content published using move (instead of copy) causes 404 due to selinux denial.CLOSED - CURRENTRELEASEjortel@redhat.comActions
Actions
Send by e-mail Copy link

Also available in: Atom PDF