Project

Profile

Help

Issue #6586

closed

pulpcore-api.service fails to start on containers running systemd

Added by mdepaulo@redhat.com almost 4 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
Installer - Moved to GitHub issues
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
CI/CD
Sprint:
Quarter:

Description

This affects the pulp_installer CI; it uses Ansible molecule to create docker containers running systemd, and installs/starts Pulp in it.

It also affects people running molecule locally for pulp_installer development.

The systemd service tasks/handlers technically succeed from Ansible's perspective, but they actually repeatedly fails to start.

journalctl _SYSTEMD_UNIT=pulpcore-api.service shows this repeating:

Apr 27 19:45:27 fedora-31 systemd[1099]: pulpcore-api.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Permission denied
Apr 27 19:45:27 fedora-31 systemd[1099]: pulpcore-api.service: Failed at step NAMESPACE spawning /usr/local/lib/pulp/bin/gunicorn: Permission denied

The issue appears to be in: pulp_installer.git/roles/pulp/templates/pulpcore-api.service.j2 These lines:

ProtectSystem=full
PrivateTmp=yes
PrivateDevices=yes

Require the container to have a higher set of capabilities, or perhaps even run privileged, to do bindmounts and possibly even namespaces. Specifying CAP_SYS_ADMIN in molecule.yml did not help.

git log & git blame revealed that these lines were added to all our services a while ago, but dropped from the other at misc points.

Possible solutions:

  1. Identify the exact set of capabilities it needs (working on this now, may not be possible)
  2. Make the container run privileged (xc94 didn't like doing this, and helped us limit privileges on containers)
  3. Drop those 3 lines. - Agreed upon on 7/31 for the sake of implementing #7259 quickly
  4. Make adding those protection lines configurable, and try to auto-detect whether the ansible managed environment (container) has capabilities. There is an ansible fact for capabilities.

Also, perhaps the systemd service tasks/handlers should have an argument to fail if the service actually fails to start?


Related issues

Blocks Pulp - Task #6585: Show the status page in pulp_installer CICLOSED - WONTFIXmdepaulo@redhat.com

Actions
Blocks Pulp - Story #6584: As a pulp_installer CI user, I see verification that pulp functions and was installed correctly at the endCLOSED - CURRENTRELEASEmdepaulo@redhat.com

Actions
Actions #1

Updated by mdepaulo@redhat.com almost 4 years ago

  • Description updated (diff)
Actions #2

Updated by mdepaulo@redhat.com almost 4 years ago

  • Description updated (diff)
Actions #3

Updated by mdepaulo@redhat.com almost 4 years ago

  • Blocks Task #6585: Show the status page in pulp_installer CI added
Actions #4

Updated by fao89 almost 4 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 71
Actions #5

Updated by mdepaulo@redhat.com almost 4 years ago

  • Description updated (diff)
Actions #6

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 71 to Sprint 72
Actions #7

Updated by bmbouter almost 4 years ago

  • Category set to Installer - Moved to GitHub issues
  • Tags deleted (Pulp 3 installer)
Actions #8

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 72 to Sprint 73
Actions #9

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 73 to Sprint 74
Actions #10

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 74 to Sprint 75
Actions #11

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 75 to Sprint 76
Actions #12

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 76 to Sprint 77
Actions #13

Updated by mdepaulo@redhat.com over 3 years ago

  • Sprint deleted (Sprint 77)

Removing from spring due to other higher priority work for me.

Actions #14

Updated by mdepaulo@redhat.com over 3 years ago

  • Blocks Story #6584: As a pulp_installer CI user, I see verification that pulp functions and was installed correctly at the end added
Actions #15

Updated by pulpbot over 3 years ago

  • Status changed from NEW to POST
Actions #16

Updated by mdepaulo@redhat.com over 3 years ago

  • Description updated (diff)
Actions #17

Updated by mdepaulo@redhat.com over 3 years ago

  • Description updated (diff)

Added by Mike DePaulo over 3 years ago

Revision e6d4cbdc | View on GitHub

Problem: pulpcore-api.service fails to start on containers running systemd

Solution: Remove systemd namespace / sandboxing features from pulpcore-api systemd unit file.

fixes: #6586 https://pulp.plan.io/issues/6586

Added by Mike DePaulo over 3 years ago

Revision e6d4cbdc | View on GitHub

Problem: pulpcore-api.service fails to start on containers running systemd

Solution: Remove systemd namespace / sandboxing features from pulpcore-api systemd unit file.

fixes: #6586 https://pulp.plan.io/issues/6586

Actions #18

Updated by Anonymous over 3 years ago

  • Status changed from POST to MODIFIED
Actions #19

Updated by ttereshc over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF