Project

Profile

Help

Issue #7867

Issue #7796 : Pulp Version 2.20 - Mongod and pulp_celerybeat failing randomly

Added by tosif.meman@kindredgroup.com 10 days ago. Updated 5 days ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
11/02/2020
Due date:
Estimated time:
Severity:
4. Urgent
Version:
2.20.0
Platform Release:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Hello Team,

We have pulp version 2.20 configured in our environment and we have faced challenges as mongodb randomly stopped working and due to that pulp_celerybeat not able to connect to the database and apparently the result is pulp_celerybeat also failing.

mongodb version - mongodb-org-server-3.2.22-1.el7.x86_64

The logs did not helped me either as logs not showing sufficient info to troubleshoot.

localhost: pulp: kombu.async.hub:ERROR: (23150-55872) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 119, in _record_heartbeat pulp: kombu.async.hub:ERROR: (23141-69152) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat pulp: kombu.async.hub:ERROR: (23240-19648) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat

MongoDB logs:

2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] db version v3.2.22 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] git version: 105acca0d443f9a47c1a5bd608fd7133840a58dd 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] allocator: tcmalloc 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] modules: none 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] build environment: 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] distmod: rhel70 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] distarch: x86_64 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] target_arch: x86_64 2020-11-04T12:19:29.375+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } } 2020-11-04T12:19:29.404+0100 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2020-11-04T12:19:29.404+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2020-11-04T12:19:29.404+0100 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2020-11-04T12:19:29.404+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),

2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2020-11-04T12:19:30.164+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2020-11-04T12:19:30.165+0100 I CONTROL [initandlisten] 2020-11-04T12:19:30.165+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 4096 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.

History

#1 Updated by bmbouter 5 days ago

  • Status changed from NEW to CLOSED - WONTFIX
  • Tags Pulp 2 added

Pulp2 is in maintenance mode so my best advice is to look at upgrading to Pulp3. Because we're not accepting new Pulp2 bugs I need to close this one.

In terms of the error message though, mongodb does not like THP enabled, and we've seen mongo issues with it before. Try disabling THP. Here are some realted docs https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/ Also see the note about it in the Pulp2 installation guide here https://docs.pulpproject.org/en/2.21/user-guide/installation/index.html#prerequisites

Please register to edit this issue

Also available in: Atom PDF