Project

Profile

Help

Story #1336

closed

As a user, it is obvious to me when the database isn't working

Added by amacdona@redhat.com over 8 years ago. Updated almost 4 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Documentation, Pulp 2
Sprint:
Quarter:

Description

At the moment, pulp failing to communicate with the db causes a number of problems. These problems are noticeable in the logs, but it is not necessarily obvious what is broken.

Example:

Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:CRITICAL: *************************************************************
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768) The Pulp server encountered an unexpected failure during initialization
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768) Traceback (most recent call last):
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)   File "/home/vagrant/devel/pulp/server/pulp/server/webservices/application.py", line 111, in wsgi_application
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     _initialize_web_services()
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)   File "/home/vagrant/devel/pulp/server/pulp/server/webservices/application.py", line 66, in _initialize_web_services
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     initialization.initialize()
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)   File "/home/vagrant/devel/pulp/server/pulp/server/initialization.py", line 31, in initialize
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     db_connection.initialize()
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)   File "/home/vagrant/devel/pulp/server/pulp/server/db/connection.py", line 53, in initialize
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768)     raise RuntimeError("The database is already initialized. It is an error to call this "
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:ERROR: (28228-60768) RuntimeError: The database is already initialized. It is an error to call this function a second time.
Oct 27 13:19:56 dev pulp[28228]: pulp.server.webservices.application:CRITICAL: *************************************************************
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536) beat raised exception <class 'pymongo.errors.AutoReconnect'>: AutoReconnect('could not connect to localhost:27017: [Errno 111] Connection refused',)
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536) Traceback (most recent call last):
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     beat.start()
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     interval = self.scheduler.tick()
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/home/vagrant/devel/pulp/server/pulp/server/async/scheduler.py", line 251, in tick
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     worker_watcher.handle_worker_heartbeat(scheduler_event)
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/home/vagrant/devel/pulp/server/pulp/server/async/worker_watcher.py", line 77, in handle_worker_heartbeat
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     worker = Worker.objects(name=event_info['worker_name']).first()
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 309, in first
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     result = queryset[0]
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 160, in __getitem__
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     return queryset._document._from_son(queryset._cursor[key],
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 595, in __getitem__
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     for doc in clone:
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1058, in next
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     if len(self.__data) or self._refresh():
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1002, in _refresh
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     self.__uuid_subtype))
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 915, in __send_message
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     res = client._send_message_with_response(message, **kwargs)
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 1192, in _send_message_with_response
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     sock_info = self.__socket(member)
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 919, in __socket
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536)     "%s %s" % (host_details, str(why)))
Oct 27 13:20:01 dev pulp[28322]: celery.beat:CRITICAL: (28322-85536) AutoReconnect: could not connect to localhost:27017: [Errno 111] Connection refused

Specifically what I dislike about this is that the only indication that mongo is not connected to pulp is that Celery cannot connect to the database. Our error seems to suggest that the database is running.

Also available in: Atom PDF