Project

Profile

Help

Story #1336

Updated by amacdona@redhat.com over 6 years ago

At the moment, pulp failing to communicate with the db causes a number of problems. These problems are noticeable in the logs, but it is not necessarily obvious what is broken.  

 Example: 

 <pre> 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:CRITICAL: ************************************************************* 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768) The Pulp server encountered an unexpected failure during initialization 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768) Traceback (most recent call last): 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     File "/home/vagrant/devel/pulp/server/pulp/server/webservices/application.py", line 111, in wsgi_application 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)       _initialize_web_services() 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     File "/home/vagrant/devel/pulp/server/pulp/server/webservices/application.py", line 66, in _initialize_web_services 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)       initialization.initialize() 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     File "/home/vagrant/devel/pulp/server/pulp/server/initialization.py", line 31, in initialize 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)       db_connection.initialize() 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     File "/home/vagrant/devel/pulp/server/pulp/server/db/connection.py", line 53, in initialize 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)       raise RuntimeError("The database is already initialized. It is an error to call this " 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768) RuntimeError: The database is already initialized. It is an error to call this function a second time. 
 Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:CRITICAL: ************************************************************* 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536) beat raised exception <class 'pymongo.errors.AutoReconnect'>: AutoReconnect('could not connect to localhost:27017: [Errno 111] Connection refused',) 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536) Traceback (most recent call last): 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       beat.start() 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       interval = self.scheduler.tick() 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/home/vagrant/devel/pulp/server/pulp/server/async/scheduler.py", line 251, in tick 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       worker_watcher.handle_worker_heartbeat(scheduler_event) 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/home/vagrant/devel/pulp/server/pulp/server/async/worker_watcher.py", line 77, in handle_worker_heartbeat 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       worker = Worker.objects(name=event_info['worker_name']).first() 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 309, in first 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       result = queryset[0] 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 160, in __getitem__ 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       return queryset._document._from_son(queryset._cursor[key], 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 595, in __getitem__ 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       for doc in clone: 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1058, in next 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       if len(self.__data) or self._refresh(): 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1002, in _refresh 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       self.__uuid_subtype)) 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 915, in __send_message 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       res = client._send_message_with_response(message, **kwargs) 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 1192, in _send_message_with_response 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       sock_info = self.__socket(member) 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 919, in __socket 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)       "%s %s" % (host_details, str(why))) 
 Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536) AutoReconnect: could not connect to localhost:27017: [Errno 111] Connection refused 

 </pre> 

 Specifically what I dislike about this is that the only indication that mongo is not connected to pulp is that Celery cannot connect to the database. Our error seems to suggest that the database is running.

Back