Pulp

Hello Team,

We have pulp version 2.20 configured in our environment and we have faced challenges as mongodb randomly stopped working and due to that pulp_celerybeat not able to connect to the database and apparently the result is pulp_celerybeat also failing.

mongodb version - mongodb-org-server-3.2.22-1.el7.x86_64

The logs did not helped me either as logs not showing sufficient info to troubleshoot.

localhost: pulp: kombu.async.hub:ERROR: (23150-55872) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 119, in _record_heartbeat pulp: kombu.async.hub:ERROR: (23141-69152) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat pulp: kombu.async.hub:ERROR: (23240-19648) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat

MongoDB logs:

2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] db version v3.2.22 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] git version: 105acca0d443f9a47c1a5bd608fd7133840a58dd 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] allocator: tcmalloc 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] modules: none 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] build environment: 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] distmod: rhel70 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] distarch: x86_64 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] target_arch: x86_64 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } } 2020-11-04T12:19:29.404+0100 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2020-11-04T12:19:29.404+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2020-11-04T12:19:29.404+0100 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2020-11-04T12:19:29.404+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),

2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2020-11-04T12:19:30.165+0100 I CONTROL [initandlisten] 2020-11-04T12:19:30.165+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 4096 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.