Project

Profile

Help

Issue #9454

closed

Pulp runs out of db connections when doing many concurrent pushes

Added by gerrod about 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Performance
Sprint:
Sprint 107
Quarter:

Description

Using master pulpcore 3.16 and master pulp_container 2.9, I run a script that pushes 50 images across 10 parallel processes of which 12 images fail to finish uploading with this error message appearing after every api call django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections. It seems that the tasks to create these images aren't always giving up their db connections as the first 38 completed fine, but the last twelve all failed (some got part way and others completely failed to start). After the script no API call was possible until Pulp had been restarted. More testing will need to be done to determine how much the concurency vs total images affects this issue.

Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [d37130e6a02a4ba2ae4f6a667981bdb7]: 127.0.0.1 - admin [27/Sep/2021:17:21:22 +0000] "HEAD /v2/python_23/blobs/sha256:d25bb576e122cde88643cec2438c57dd098a27634e>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [4536f0c6a9fb46fc9a9d49f066cc6556]: django.request:ERROR: Internal Server Error: /v2/python_23/blobs/sha256:ba5a5fe4330168081f2a7e46f72c7552456d4d604ad27feadb>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: psycopg2.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: The above exception was the direct cause of the following exception:
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = get_response(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return view_func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.dispatch(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = self.handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 287, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = super().handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.raise_uncaught_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise exc
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 497, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.initial(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 414, in initial
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.perform_authentication(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 324, in perform_authentication
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     request.user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 227, in user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._authenticate()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*func_args, **func_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/__init__.py", line 76, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = backend.authenticate(request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/backends.py", line 42, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = UserModel._default_manager.get_by_natural_key(username)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.get(**{self.model.USERNAME_FIELD: username})
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     num = len(clone)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._fetch_all()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._result_cache = list(self._iterable_class(self))
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     cursor = self.connection.cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self._cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.ensure_connection()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Related issues

Copied to Container Support - Issue #9522: Pulp runs out of db connections when doing many concurrent pushesCLOSED - CURRENTRELEASEipanova@redhat.comActions
Copied to Container Support - Backport #9538: Backport to 2.9: Pulp runs out of db connections when doing many concurrent pushesCLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Actions #1

Updated by ipanova@redhat.com about 3 years ago

When this PR was introduced in core https://github.com/pulp/pulpcore/commit/91a57d3e6efd33a0ce8ba0544f2f59d60692360f#diff-453fdf3a6e81f92ff5391cf65f9c6154852133a656d82082021e3beaa887776f according changes to the sub-classed Container Registry Handler were not made.

Django 3.2 opens up a new connection to the DB for each co-routine that accesses the DB. All ORM interactions need to be wrapped in 'asgiref.sync.sync_to_async' in order to ensure that all DB queries are executed using a single connection in a serial manner. Also see for reference https://github.com/pulp/pulpcore/pull/1678/files

Actions #2

Updated by ipanova@redhat.com about 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 107
Actions #3

Updated by pulpbot about 3 years ago

  • Status changed from NEW to POST
Actions #4

Updated by mdellweg about 3 years ago

  • Assignee set to mdellweg

Added by mdellweg about 3 years ago

Revision 937f0812 | View on GitHub

Use sync_to_async for orm calls in the content app

fixes #9454

Added by mdellweg about 3 years ago

Revision 937f0812 | View on GitHub

Use sync_to_async for orm calls in the content app

fixes #9454

Actions #5

Updated by mdellweg about 3 years ago

  • Status changed from POST to MODIFIED
Actions #6

Updated by ipanova@redhat.com about 3 years ago

  • Copied to Issue #9522: Pulp runs out of db connections when doing many concurrent pushes added
Actions #7

Updated by rafzei about 3 years ago

Just faced this issue when uploading tags to pulp_container. I hope it will be released soon.

Actions #8

Updated by ipanova@redhat.com about 3 years ago

  • Copied to Backport #9538: Backport to 2.9: Pulp runs out of db connections when doing many concurrent pushes added
Actions #9

Updated by ipanova@redhat.com almost 3 years ago

  • Sprint/Milestone set to 2.10.0
Actions #10

Updated by pulpbot almost 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF