Pulp

The failing component is likely long-term reconnect with qpidd because there is another error with pulp_celerybeat that occurs if mongod is not available [0]. That other error is experienced on a much shorter timeline (minutes) with a different traceback. By deduction I believe this BZ is likely caused by reconnect support with respect to qpidd.

[0]: https://bugzilla.redhat.com/show_bug.cgi?id=1129488

+ This comment was cloned from Bugzilla #1118404 comment 1 +

The recommended workaround for this is two-fold. First, avoid getting into this situation by not having qpidd unavailable from pulp_celerybeat for hours/days. Second, if you do experience the issue you can recover by restarting pulp_celerybeat. pulp_celerybeat should be perfectly save to restart at any time.

Unable to reproduce with either qpid.messaging in isolation. The process is leaking file descriptors opened on pipes (not sockets).
The Transport in kombu.transport.qpid opens a pipe in its constructor which (best I can tell) is never closed.

Still investigating.

Fixing in kombu.

https://github.com/celery/kombu/issues/476
https://github.com/celery/kombu/pull/477

Then, patch in pulp/deps.

QE, do the following to reproduce:

0) Install python-kombu 3.0.24-7.pulp (this should come with pulp 2.6.2)
1) Leave all pulp_* and httpd services off
2) ensure qpidd and mongodb are running
3) start pulp_celerybeat and verify in the log that it starts ok
4) stop qpidd
5) observe the log statements that show pulp_celerybeat is trying to reconnect
6) Determine the process id of pulp_celerybeat, say it's 12345
7) Use lsof to count the number of pipes by running:

sudo lsof -p 12345 | grep pipe | wc

8) Let pulp_celerybeat attempt to reconnect over say 5-10 minutes. pulp_celerybeat should not have been restarted since step 3
9) Use lsof again after the 5-10 minutes and verify the first number is the same. It was 6 on my machine when I did this.

Verified

[root@mgmt12 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@mgmt12 ~]# 

[root@mgmt12 ~]# service qpidd stop
Stopping Qpid AMQP daemon: [  OK  ]
[root@mgmt12 ~]# 
[root@mgmt12 ~]# 
[root@mgmt12 ~]# service pulp_celerybeat status
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
pulp_celerybeat (pid 818) is running.
[root@mgmt12 ~]# sudo lsof -p 818 | grep pipe | wc
      6      54     432
[root@mgmt12 ~]# 
[root@mgmt12 ~]# 
[root@mgmt12 ~]# sudo lsof -p 818 | grep pipe | wc
      6      54     432
[root@mgmt12 ~]#