Pulp

This affects the pulp_installer CI; it uses Ansible molecule to create docker containers running systemd, and installs/starts Pulp in it.

It also affects people running molecule locally for pulp_installer development.

The systemd service tasks/handlers technically succeed from Ansible's perspective, but they actually repeatedly fails to start.

journalctl _SYSTEMD_UNIT=pulpcore-api.service shows this repeating:

Apr 27 19:45:27 fedora-31 systemd[1099]: pulpcore-api.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Permission denied
Apr 27 19:45:27 fedora-31 systemd[1099]: pulpcore-api.service: Failed at step NAMESPACE spawning /usr/local/lib/pulp/bin/gunicorn: Permission denied

The issue appears to be in: pulp_installer.git/roles/pulp/templates/pulpcore-api.service.j2 These lines:

ProtectSystem=full
PrivateTmp=yes
PrivateDevices=yes

Require the container to have a higher set of capabilities, or perhaps even run privileged, to do bindmounts and possibly even namespaces. Specifying CAP_SYS_ADMIN in molecule.yml did not help.

git log & git blame revealed that these lines were added to all our services a while ago, but dropped from the other at misc points.

Possible solutions:

1. Identify the exact set of capabilities it needs (working on this now, may not be possible)
2. Make the container run privileged (xc94 didn't like doing this, and helped us limit privileges on containers)
3. Drop those 3 lines. - Agreed upon on 7/31 for the sake of implementing #7259 quickly
4. Make adding those protection lines configurable, and try to auto-detect whether the ansible managed environment (container) has capabilities. There is an ansible fact for capabilities.

Also, perhaps the systemd service tasks/handlers should have an argument to fail if the service actually fails to start?