Pulp

sudo systemctl start pulp_workers                 # Start pulp_workers with systemd (you can verify they run with ps -awfux | grep celery)
sudo pkill -9 -f reserved_resource_worker         # Stop the workers outside of systemd (or stop each worker individually with systemctl)
sudo systemctl start pulp_workers                # Start the workers again with systemd
ps -awfux | grep celery                          # Note that no workers are running

This is an issue with how we chain-load the workers.

The relevant contents of the pulp_workers.service file are the following.

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/bin/python -m  pulp.server.async.manage_workers start
ExecStop=/usr/bin/python -m  pulp.server.async.manage_workers stop

When started and stopped, this service will execute a call a Python utility with a command line option of "start" or "stop", respectively. This utility, located in pulp/server/pulp/server/async/manage_workers.py, will start or stop the workers by writing out the individual worker unit files to /run/systemd/system/ and using Popen to start/stop them all via systemctl. The utility then exits.

Because "RemainAfterExit" is set to "True", the pulp_workers.service unit will always have an "active" status after being started, regardless of what happens to the workers thereafter. "systemctl start" does not do anything if the unit is already active, hence our problem.

The way to work around this would be to use systemctl stop pulp_workers prior to starting them, or else to use systemctl restart pulp_workers.