Project

Profile

Help

Issue #1564

Updated by bmbouter about 7 years ago

After executing a "heavy" workload, the resource_manager queue may have unacked messages in it. To reproduce you can: 

 1. Execute the heavy workload and let it finish. For instance you could sync many repos and publish them. 

 2. List the Qpid queue statistics with: 

 <pre> 
 qpid-stat -q 
 </pre> 


 3. Observe a output like the following: 

 <pre> 
 queue                                                                               dur    autoDel    excl    msg     msgIn    msgOut    bytes    bytesIn    bytesOut    cons    bind  
 ================================================================================================================================================================= 
 <snip> 
 resource_manager                                                                    Y                       19     338      319      33.4k     567k       534k          1       2 
 <snip> 
 </pre> 

 4. Observe that all Pulp tasks have completed with state finished, and yet there are still messages in the 'msg' column. 19 in the example above. You can verify all Pulp tasks as being completed by connecting to mongo, switching to the pulp_database, and running the following command. If all pulp tasks are finished that query will return 0. 

 <pre> 
 db.task_status.find({ state:{ $ne: "finished" } } ).count() 
 </pre> 

 Note: This has been observed on both EL6 and EL7, and it may be observable on Fedoras also. 

 Any of the unacked messages will be "rerun" when the resource_manager restarts because the message broker will redeliver them. This is a user noticeable defect.

Back