Project

Profile

Help

Backport #9538

Backport to 2.9: Pulp runs out of db connections when doing many concurrent pushes

Added by ipanova@redhat.com about 1 month ago. Updated 4 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Triaged:
Yes
Sprint Candidate:
No
Tags:
Performance
Sprint:
Sprint 110
Quarter:

Description

Using master pulpcore 3.16 and master pulp_container 2.9, I run a script that pushes 50 images across 10 parallel processes of which 12 images fail to finish uploading with this error message appearing after every api call django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections. It seems that the tasks to create these images aren't always giving up their db connections as the first 38 completed fine, but the last twelve all failed (some got part way and others completely failed to start). After the script no API call was possible until Pulp had been restarted. More testing will need to be done to determine how much the concurency vs total images affects this issue.

Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [d37130e6a02a4ba2ae4f6a667981bdb7]: 127.0.0.1 - admin [27/Sep/2021:17:21:22 +0000] "HEAD /v2/python_23/blobs/sha256:d25bb576e122cde88643cec2438c57dd098a27634e>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [4536f0c6a9fb46fc9a9d49f066cc6556]: django.request:ERROR: Internal Server Error: /v2/python_23/blobs/sha256:ba5a5fe4330168081f2a7e46f72c7552456d4d604ad27feadb>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: psycopg2.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: The above exception was the direct cause of the following exception:
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = get_response(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return view_func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.dispatch(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = self.handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 287, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = super().handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.raise_uncaught_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise exc
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 497, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.initial(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 414, in initial
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.perform_authentication(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 324, in perform_authentication
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     request.user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 227, in user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._authenticate()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*func_args, **func_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/__init__.py", line 76, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = backend.authenticate(request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/backends.py", line 42, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = UserModel._default_manager.get_by_natural_key(username)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.get(**{self.model.USERNAME_FIELD: username})
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     num = len(clone)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._fetch_all()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._result_cache = list(self._iterable_class(self))
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     cursor = self.connection.cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self._cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.ensure_connection()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Related issues

Copied from Container Support - Issue #9454: Pulp runs out of db connections when doing many concurrent pushesMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 6ec1c4e5 View on GitHub
Added by mdellweg 4 days ago

Use sync_to_async for orm calls in the content app

backports #9454 closes #9538 (cherry picked from commit 937f0812ab0bed258755d7fd1846ead651861dd8)

Revision 6ec1c4e5 View on GitHub
Added by mdellweg 4 days ago

Use sync_to_async for orm calls in the content app

backports #9454 closes #9538 (cherry picked from commit 937f0812ab0bed258755d7fd1846ead651861dd8)

History

#1 Updated by ipanova@redhat.com about 1 month ago

  • Copied from Issue #9454: Pulp runs out of db connections when doing many concurrent pushes added

#2 Updated by ipanova@redhat.com about 1 month ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#3 Updated by pulpbot about 1 month ago

  • Status changed from ASSIGNED to POST

#4 Updated by ipanova@redhat.com about 1 month ago

  • Sprint/Milestone set to 2.9.1

#5 Updated by rchan 23 days ago

  • Sprint changed from Sprint 108 to Sprint 109

#6 Updated by rchan 8 days ago

  • Sprint changed from Sprint 109 to Sprint 110

#7 Updated by mdellweg 4 days ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by pulpbot 4 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF