Project

Profile

Help

Issue #9515

content app doesn't survive PostgreSQL disconnect in pulpcore 3.14.7

Added by evgeni about 1 month ago. Updated 22 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 109
Quarter:

Description

Ohai,

on pulpcore 3.14 (I didn't try newer yet), when the DB connection between pulpcore-content and PostgreSQL dies (e.g. because PostgreSQL was restarted), the content app seems not to notice this, and serves 500 errors:

pulpcore-content[659]: pulp [None]: asyncio:ERROR: Task exception was never retrieved
pulpcore-content[659]: future:  exception=OperationalError('terminating connection due to administrator command\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)>
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: The above exception was the direct cause of the following exception:
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 45, in _heartbeat
pulpcore-content[659]: content_app_status, created = await loop.run_in_executor(None, get_status_blocking)
pulpcore-content[659]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
pulpcore-content[659]: result = self.fn(*self.args, **self.kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 42, in get_status_blocking
pulpcore-content[659]: return ContentAppStatus.objects.get_or_create(name=name)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-content[659]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
pulpcore-content[659]: return self.get(**kwargs), False
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
pulpcore-content[659]: num = len(clone)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-content[659]: self._fetch_all()
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-content[659]: self._result_cache = list(self._iterable_class(self))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
pulpcore-content[659]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-content[659]: cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-content[659]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-content[659]: return executor(sql, params, many, context)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-content[659]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: django.db.utils.OperationalError: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: pulp [None]: asyncio:ERROR: Task exception was never retrieved
pulpcore-content[659]: future:  exception=OperationalError('terminating connection due to administrator command\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)>
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: The above exception was the direct cause of the following exception:
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 45, in _heartbeat
pulpcore-content[659]: content_app_status, created = await loop.run_in_executor(None, get_status_blocking)
pulpcore-content[659]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
pulpcore-content[659]: result = self.fn(*self.args, **self.kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 42, in get_status_blocking
pulpcore-content[659]: return ContentAppStatus.objects.get_or_create(name=name)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-content[659]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
pulpcore-content[659]: return self.get(**kwargs), False
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
pulpcore-content[659]: num = len(clone)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-content[659]: self._fetch_all()
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-content[659]: self._result_cache = list(self._iterable_class(self))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
pulpcore-content[659]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-content[659]: cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-content[659]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-content[659]: return executor(sql, params, many, context)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-content[659]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: django.db.utils.OperationalError: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: pulp [None]: asyncio:ERROR: Task exception was never retrieved
pulpcore-content[659]: future:  exception=OperationalError('terminating connection due to administrator command\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)>
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: The above exception was the direct cause of the following exception:
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 45, in _heartbeat
pulpcore-content[659]: content_app_status, created = await loop.run_in_executor(None, get_status_blocking)
pulpcore-content[659]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
pulpcore-content[659]: result = self.fn(*self.args, **self.kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 42, in get_status_blocking
pulpcore-content[659]: return ContentAppStatus.objects.get_or_create(name=name)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-content[659]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
pulpcore-content[659]: return self.get(**kwargs), False
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
pulpcore-content[659]: num = len(clone)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-content[659]: self._fetch_all()
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-content[659]: self._result_cache = list(self._iterable_class(self))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
pulpcore-content[659]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-content[659]: cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-content[659]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-content[659]: return executor(sql, params, many, context)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-content[659]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: django.db.utils.OperationalError: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: pulp [None]: asyncio:ERROR: Task exception was never retrieved
pulpcore-content[659]: future:  exception=OperationalError('terminating connection due to administrator command\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)>
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: The above exception was the direct cause of the following exception:
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 45, in _heartbeat
pulpcore-content[659]: content_app_status, created = await loop.run_in_executor(None, get_status_blocking)
pulpcore-content[659]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
pulpcore-content[659]: result = self.fn(*self.args, **self.kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 42, in get_status_blocking
pulpcore-content[659]: return ContentAppStatus.objects.get_or_create(name=name)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-content[659]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
pulpcore-content[659]: return self.get(**kwargs), False
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
pulpcore-content[659]: num = len(clone)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-content[659]: self._fetch_all()
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-content[659]: self._result_cache = list(self._iterable_class(self))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
pulpcore-content[659]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-content[659]: cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-content[659]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-content[659]: return executor(sql, params, many, context)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-content[659]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: django.db.utils.OperationalError: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: pulp [None]: asyncio:ERROR: Task exception was never retrieved
pulpcore-content[659]: future:  exception=OperationalError('terminating connection due to administrator command\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)>
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: The above exception was the direct cause of the following exception:
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 45, in _heartbeat
pulpcore-content[659]: content_app_status, created = await loop.run_in_executor(None, get_status_blocking)
pulpcore-content[659]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
pulpcore-content[659]: result = self.fn(*self.args, **self.kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 42, in get_status_blocking
pulpcore-content[659]: return ContentAppStatus.objects.get_or_create(name=name)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-content[659]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
pulpcore-content[659]: return self.get(**kwargs), False
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
pulpcore-content[659]: num = len(clone)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-content[659]: self._fetch_all()
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-content[659]: self._result_cache = list(self._iterable_class(self))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
pulpcore-content[659]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-content[659]: cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-content[659]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-content[659]: return executor(sql, params, many, context)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-content[659]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: django.db.utils.OperationalError: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: [2021-10-18 10:18:25 +0000] [1138] [ERROR] Error handling request
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: The above exception was the direct cause of the following exception:
pulpcore-content[659]: Traceback (most recent call last):
pulpcore-content[659]: File "/usr/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
pulpcore-content[659]: resp = await self._request_handler(request)
pulpcore-content[659]: File "/usr/lib64/python3.6/site-packages/aiohttp/web_app.py", line 499, in _handle
pulpcore-content[659]: resp = await handler(request)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/handler.py", line 142, in list_distributions
pulpcore-content[659]: base_paths = await loop.run_in_executor(None, get_base_paths_blocking)
pulpcore-content[659]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
pulpcore-content[659]: result = self.fn(*self.args, **self.kwargs)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/pulpcore/content/handler.py", line 135, in get_base_paths_blocking
pulpcore-content[659]: base_paths = list(Distribution.objects.values_list("base_path", flat=True))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
pulpcore-content[659]: self._fetch_all()
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-content[659]: self._result_cache = list(self._iterable_class(self))
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 182, in __iter__
pulpcore-content[659]: for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1094, in results_iter
pulpcore-content[659]: results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-content[659]: cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-content[659]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-content[659]: return executor(sql, params, many, context)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-content[659]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-content[659]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-content[659]: return self.cursor.execute(sql, params)
pulpcore-content[659]: django.db.utils.OperationalError: terminating connection due to administrator command
pulpcore-content[659]: server closed the connection unexpectedly
pulpcore-content[659]: This probably means the server terminated abnormally
pulpcore-content[659]: before or while processing the request.
pulpcore-content[659]: [18/Oct/2021:10:18:25 +0000] "GET /pulp/content/ HTTP/1.1" 500 244 "-" "curl/7.29.0"

Related issues

Related to Pulp - Issue #9276: Content app can have unusable/closed db connections in pulpcore 3.15/3.16MODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 56f5e381 View on GitHub
Added by dkliban@redhat.com 22 days ago

Improves tolerance of db connection failures in pulpcore-content

This patch moves Handler._reset_db_connection() call to get_base_paths_blocking(). This ensures that the connection is reset inside the same thread that is used to perform ORM calls. This fixes a bug where the /pulp/content/ page would return a 500 error after the connection the database is closed due to a network problem or a database restart.

A call to Handler._reset_db_connection() has also been added to the _heartbeat() co-routine. This ensures that the heartbeat continues to be recorded in the database even if the db connection was closed.

fixes: #9515 https://pulp.plan.io/issues/9515

History

#1 Updated by evgeni about 1 month ago

This is correctly detected by the API in status:

    "online_content_apps": [],

#2 Updated by evgeni about 1 month ago

You'd think it's a dupe of https://pulp.plan.io/issues/9275

But this is on python3-pulpcore-3.14.7-1.el7.noarch, which got the above backported in https://pulp.plan.io/issues/9282

#3 Updated by evgeni about 1 month ago

And certainly related to https://pulp.plan.io/issues/9276, no idea how you want to stack those (or dupe ;))

#4 Updated by evgeni about 1 month ago

Just verified, this also happens with pulpcore 3.16 (running on Python 3.8.11, Django 3.2.8)

#6 Updated by evgeni about 1 month ago

The following patch makes the content app work for me again after a DB restart, but I think using _reset_db_connection is just papering over the actual issue.

commit 60a21b41da1acd4f0bd6fc1984d8b66773d1ff3c
Author: Evgeni Golov 
Date:   Tue Oct 19 09:16:29 2021 +0200

    reset DB connection before trying to get status

diff --git pulpcore/content/__init__.py pulpcore/content/__init__.py
index 396db3a1c..623098d77 100644
--- pulpcore/content/__init__.py
+++ pulpcore/content/__init__.py
@@ -40,6 +40,7 @@ async def _heartbeat():
     msg = i8ln_msg.format(name=name, interarrival=heartbeat_interval)
 
     def get_status_blocking():
+        Handler._reset_db_connection()
         return ContentAppStatus.objects.get_or_create(name=name)
 
     while True:

#7 Updated by ipanova@redhat.com about 1 month ago

  • Status changed from NEW to CLOSED - DUPLICATE

#8 Updated by mdellweg about 1 month ago

  • Is duplicate of Issue #9276: Content app can have unusable/closed db connections in pulpcore 3.15/3.16 added

#9 Updated by ttereshc about 1 month ago

  • Status changed from CLOSED - DUPLICATE to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Priority changed from Normal to Urgent

Reopening the issue to track changes needed for 3.14.z. Content app changed significantly between 3.15+ and 3.14, the solution for 3.15+ is different than the one needed for 3.14.z.

#10 Updated by ttereshc about 1 month ago

  • Is duplicate of deleted (Issue #9276: Content app can have unusable/closed db connections in pulpcore 3.15/3.16)

#11 Updated by ttereshc about 1 month ago

  • Related to Issue #9276: Content app can have unusable/closed db connections in pulpcore 3.15/3.16 added

#12 Updated by ttereshc about 1 month ago

  • Sprint set to Sprint 108

#13 Updated by ttereshc about 1 month ago

  • Subject changed from content app doesn't survive PostgreSQL disconnect to content app doesn't survive PostgreSQL disconnect in pulpcore 3.14.7
  • Sprint deleted (Sprint 108)

#14 Updated by ttereshc about 1 month ago

  • Sprint set to Sprint 108

#15 Updated by ttereshc about 1 month ago

  • Tags Katello added

#16 Updated by dkliban@redhat.com about 1 month ago

  • Triaged changed from No to Yes

#17 Updated by pulpbot about 1 month ago

  • Status changed from ASSIGNED to POST

#18 Updated by rchan 23 days ago

  • Sprint changed from Sprint 108 to Sprint 109

#19 Updated by dkliban@redhat.com 22 days ago

  • Status changed from POST to MODIFIED

#20 Updated by pulpbot 22 days ago

  • Sprint/Milestone set to 3.14.9

#21 Updated by pulpbot 22 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF