This seems to produce stuck waiting tasks for me:

export I=$RANDOM

http POST $BASE_ADDR/pulp/api/v3/remotes/file/file/ name=foo$I url=bar                                                   
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/file/file/ | jq -r '.results[0] | .pulp_href')

sudo systemctl stop redis
http PATCH :$REMOTE_HREF name=test$I
sudo systemctl start redis

# this task will be stuck forever and all future tasks get stuck too
http PATCH :$REMOTE_HREF name=test

Pasting this one comment from github for historical record:

I might know what it is. The task is spawned onto a queue with the name of the worker they are assigned to. This worker is determined via _acquire_worker with a database query. When redis goes down, all the workers get killed off (and recreated), but their records remain in the database for 30 seconds until they get cleaned up in the background.

If it happens quickly enough, it could try to assign the task to a dead worker, and dump the task into a queue for the dead worker. That should be easy enough to verify...

This would have been introduced when we started using a random(ish?) number component in the worker names.

Setting the priority to high since Ansible also reported this.

PR: https://github.com/pulp/pulpcore/pull/891