Container Support

Using master pulpcore 3.16 and master pulp_container 2.9, I run a script that pushes 50 images across 10 parallel processes of which 12 images fail to finish uploading with this error message appearing after every api call django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections. It seems that the tasks to create these images aren't always giving up their db connections as the first 38 completed fine, but the last twelve all failed (some got part way and others completely failed to start). After the script no API call was possible until Pulp had been restarted. More testing will need to be done to determine how much the concurency vs total images affects this issue.

Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [d37130e6a02a4ba2ae4f6a667981bdb7]: 127.0.0.1 - admin [27/Sep/2021:17:21:22 +0000] "HEAD /v2/python_23/blobs/sha256:d25bb576e122cde88643cec2438c57dd098a27634e>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: pulp [4536f0c6a9fb46fc9a9d49f066cc6556]: django.request:ERROR: Internal Server Error: /v2/python_23/blobs/sha256:ba5a5fe4330168081f2a7e46f72c7552456d4d604ad27feadb>
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: psycopg2.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: The above exception was the direct cause of the following exception:
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: Traceback (most recent call last):
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = get_response(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return view_func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.dispatch(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = self.handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 287, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     response = super().handle_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.raise_uncaught_exception(exc)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise exc
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 497, in dispatch
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.initial(request, *args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 414, in initial
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.perform_authentication(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 324, in perform_authentication
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     request.user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 227, in user
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._authenticate()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/request.py", line 380, in _authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user_auth_tuple = authenticator.authenticate(self)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/token_verification.py", line 64, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return super().authenticate(request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 87, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.authenticate_credentials(userid, password, request)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = authenticate(request=request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*func_args, **func_kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/__init__.py", line 76, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = backend.authenticate(request, **credentials)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/backends.py", line 42, in authenticate
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     user = UserModel._default_manager.get_by_natural_key(username)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self.get(**{self.model.USERNAME_FIELD: username})
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     num = len(clone)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._fetch_all()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self._result_cache = list(self._iterable_class(self))
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     cursor = self.connection.cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return self._cursor()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.ensure_connection()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connect()
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     self.connection = self.get_new_connection(conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     return func(*args, **kwargs)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     connection = Database.connect(**conn_params)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Sep 27 17:21:22 pulp3-source-fedora34.fedora.example.com gunicorn[85663]: django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections