Project

Profile

Help

Issue #9522

Pulp runs out of db connections when doing many concurrent pushes

Added by ipanova@redhat.com about 1 month ago. Updated 4 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Performance
Sprint:
Sprint 110
Quarter:

Description

Using master pulpcore 3.16 and master pulp_container 2.9, I run a script that pushes 50 images across 10 parallel processes of which 12 images fail to finish uploading with this error message appearing after every api call django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections. It seems that the tasks to create these images aren't always giving up their db connections as the first 38 completed fine, but the last twelve all failed (some got part way and others completely failed to start). After the script no API call was possible until Pulp had been restarted. More testing will need to be done to determine how much the concurency vs total images affects this issue.

Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [d37130e6a02a4ba2ae4f6a667981bdb7]: 127.0.0.1 - admin [27/Sep/2021:17:21:22 +0000] "HEAD /v2/python_23/blobs/sha256:d25bb576e122cde88643cec2438c57dd098a27634e>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [4536f0c6a9fb46fc9a9d49f066cc6556]: django.request:ERROR: Internal Server Error: /v2/python_23/blobs/sha256:ba5a5fe4330168081f2a7e46f72c7552456d4d604ad27feadb>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: psycopg2.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: The above exception was the direct cause of the following exception:
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = get_response(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return view_func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.dispatch(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = self.handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 287, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = super().handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.raise_uncaught_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise exc
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 497, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.initial(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 414, in initial
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.perform_authentication(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 324, in perform_authentication
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     request.user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 227, in user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._authenticate()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*func_args, **func_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/__init__.py", line 76, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = backend.authenticate(request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/backends.py", line 42, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = UserModel._default_manager.get_by_natural_key(username)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.get(**{self.model.USERNAME_FIELD: username})
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     num = len(clone)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._fetch_all()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._result_cache = list(self._iterable_class(self))
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     cursor = self.connection.cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self._cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.ensure_connection()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Related issues

Copied from Container Support - Issue #9454: Pulp runs out of db connections when doing many concurrent pushesMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 1661ce7b View on GitHub
Added by mdellweg 4 days ago

Use loop.run_in_executor() for orm calls in the content app closes #9522

Revision 1661ce7b View on GitHub
Added by mdellweg 4 days ago

Use loop.run_in_executor() for orm calls in the content app closes #9522

History

#1 Updated by ipanova@redhat.com about 1 month ago

  • Copied from Issue #9454: Pulp runs out of db connections when doing many concurrent pushes added

#2 Updated by ipanova@redhat.com about 1 month ago

  • Subject changed from Backport to 2.9: Pulp runs out of db connections when doing many concurrent pushes to Backport to 2.8: Pulp runs out of db connections when doing many concurrent pushes

#3 Updated by ipanova@redhat.com about 1 month ago

  • Sprint/Milestone set to 2.8.2

#4 Updated by rchan about 1 month ago

  • Sprint changed from Sprint 107 to Sprint 108

#5 Updated by ipanova@redhat.com about 1 month ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#6 Updated by ipanova@redhat.com about 1 month ago

  • Tracker changed from Backport to Issue
  • Subject changed from Backport to 2.8: Pulp runs out of db connections when doing many concurrent pushes to Pulp runs out of db connections when doing many concurrent pushes
  • Severity set to 2. Medium
  • Groomed set to No

#7 Updated by rchan 23 days ago

  • Sprint changed from Sprint 108 to Sprint 109

#9 Updated by rchan 8 days ago

  • Sprint changed from Sprint 109 to Sprint 110

#10 Updated by mdellweg 4 days ago

  • Status changed from POST to MODIFIED

#11 Updated by pulpbot 4 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF