Pulp

It looks like tasks are currently being requeued for Celery 4 even if task_reject_on_worker_lost is not set. They shouldn't be and we need to investigate and potentially open an issue against Celery.

So I was able to reproduce the behavior in standalone celery where messages are persisted on warm shutdown even if task_reject_on_worker_lost is not set. It turns out that if you run pkill -f celery instead of kill $CHILD_PROCESS_ID, the message gets persisted.

This is why when shutting down pulp_resource_manager via systemctl, we're seeing messages getting persisted. It's killing (or doing a warm shutdown) on both processes. I have no idea why this is. I can open an upstream celery issue but this behavior sounds pretty much the same as some existing bugs [1][2].

Also, the message persisting is not a problem for us though. We're concerned about messages being lost and if a message gets persisted, it simply runs the next time pulp starts up. We're not concerned here about double execution either since pulp_workers are not running.

[1] https://github.com/celery/celery/issues/3802
[2] https://github.com/celery/celery/issues/3796

So basically we just need to enable task_reject_on_worker_lost and we're good to go.

Opened separate issue for Pulp 3:

https://pulp.plan.io/issues/2958

I would probably recommend using the following workflow for testing as it's a bit more precise in that it only kill the child worker process. Using sudo systemctl restart pulp_resource_manager will kill both the child and the parent which will potentially leave the message in the queue and thus would be a false positive.

sudo systemctl stop pulp_workers # may need to wait 30 seconds for this to die
pulp-admin rpm repo sync run --repo-id zoo --bg
qpid-stat -q # observe that the queue depth of the resource_manager queue is 1
ps auxf | grep resource_manager # grab the child process id (e.g. 12345) 
sudo kill 12345
qpid-stat -q # observe that the queue depth of the resource_manager queue is still 1
sudo systemctl restart pulp_resource_manager
sudo systemctl start pulp_workers # may need to wait 30 seconds for this to start and pick up task
pulp-admin tasks list -s waiting # should be empty