Project

Profile

Help

Issue #2958

Updated by daviddavis over 6 years ago

This is a clone of #2954 for pulp 3. 

 From #2954: 

 The resource_manager queue loses a currently running _queue_reserved_task if the resource manager is restarted with <code>sudo systemctl restart pulp_resource_manager</code>. 

 The task is lost from the queue but still has an incorrect TaskStatus record showing as waiting which will never run. 

 Note that if you <code>sudo pkill -9 -f resource_manager</code> and the <code>sudo systemctl start pulp_resource_manager</code> it does not lose the task. 

 <pre> 
 sudo systemctl stop pulp_workers 
 pulp-admin rpm repo sync run --repo-id zoo 
 qpid-stat -q                          <<-- observe that the queue depth of the resource_manager queue is 1 
 sudo systemctl restart pulp_resource_manager 
 qpid-stat -q                          <<-- observe that the queue depth of the resource_manager queue is 0 
 pulp-admin tasks list -s waiting      <<-- observe that the task which is gone is listed as 'waiting', but it will never run because it is gone 
 </pre> 

 We need to make sure that this doesn't happen in Celery 4. There's a config task that should prevent this: 

 http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-reject-on-worker-lost 

Back