Issue #9454
closedPulp runs out of db connections when doing many concurrent pushes
Description
Using master pulpcore 3.16 and master pulp_container 2.9, I run a script that pushes 50 images across 10 parallel processes of which 12 images fail to finish uploading with this error message appearing after every api call django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
. It seems that the tasks to create these images aren't always giving up their db connections as the first 38 completed fine, but the last twelve all failed (some got part way and others completely failed to start). After the script no API call was possible until Pulp had been restarted. More testing will need to be done to determine how much the concurency vs total images affects this issue.
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [d37130e6a02a4ba2ae4f6a667981bdb7]: 127.0.0.1 - admin [27/Sep/2021:17:21:22 +0000] "HEAD /v2/python_23/blobs/sha256:d25bb576e122cde88643cec2438c57dd098a27634e>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [4536f0c6a9fb46fc9a9d49f066cc6556]: django.request:ERROR: Internal Server Error: /v2/python_23/blobs/sha256:ba5a5fe4330168081f2a7e46f72c7552456d4d604ad27feadb>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: psycopg2.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: The above exception was the direct cause of the following exception:
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: response = get_response(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return view_func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return self.dispatch(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: response = self.handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 287, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: response = super().handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.raise_uncaught_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: raise exc
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 497, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.initial(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 414, in initial
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.perform_authentication(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 324, in perform_authentication
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: request.user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 227, in user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self._authenticate()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*func_args, **func_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/__init__.py", line 76, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: user = backend.authenticate(request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/backends.py", line 42, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: user = UserModel._default_manager.get_by_natural_key(username)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return self.get(**{self.model.USERNAME_FIELD: username})
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: num = len(clone)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self._fetch_all()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self._result_cache = list(self._iterable_class(self))
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: cursor = self.connection.cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return self._cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.ensure_connection()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
Related issues
Updated by ipanova@redhat.com almost 3 years ago
When this PR was introduced in core https://github.com/pulp/pulpcore/commit/91a57d3e6efd33a0ce8ba0544f2f59d60692360f#diff-453fdf3a6e81f92ff5391cf65f9c6154852133a656d82082021e3beaa887776f according changes to the sub-classed Container Registry Handler were not made.
Django 3.2 opens up a new connection to the DB for each co-routine that accesses the DB. All ORM interactions need to be wrapped in 'asgiref.sync.sync_to_async' in order to ensure that all DB queries are executed using a single connection in a serial manner. Also see for reference https://github.com/pulp/pulpcore/pull/1678/files
Updated by ipanova@redhat.com almost 3 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 107
Updated by pulpbot almost 3 years ago
- Status changed from NEW to POST
Added by mdellweg almost 3 years ago
Added by mdellweg almost 3 years ago
Revision 937f0812 | View on GitHub
Use sync_to_async for orm calls in the content app
fixes #9454
Updated by mdellweg almost 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset 937f0812ab0bed258755d7fd1846ead651861dd8.
Updated by ipanova@redhat.com almost 3 years ago
- Copied to Issue #9522: Pulp runs out of db connections when doing many concurrent pushes added
Updated by rafzei almost 3 years ago
Just faced this issue when uploading tags to pulp_container. I hope it will be released soon.
Updated by ipanova@redhat.com almost 3 years ago
- Copied to Backport #9538: Backport to 2.9: Pulp runs out of db connections when doing many concurrent pushes added
Updated by pulpbot over 2 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Use sync_to_async for orm calls in the content app
fixes #9454