Actions
Issue #5109
closed
500 - illegal-argument: Value for replyText is too large(320)
Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Description
Hi,
since some days I got the following error when I try to delete or Sync a Repo with pulp-admin:
[14:13:26 root@repos]{/var/log/mongodb}>pulp-admin rpm repo sync run --force-full --repo-id=jb-centos7-mariadb-main
+----------------------------------------------------------------------+
Synchronizing Repository [jb-centos7-mariadb-main]
+----------------------------------------------------------------------+
The following sync configuration options will be used:
Force Full: True
An internal error occurred on the Pulp server:
RequestException: POST request
on /pulp/api/v2/repositories/jb-centos7-mariadb-main/actions/sync/ failed with
500 - illegal-argument: Value for replyText is too large(320)
I got this error for every project!
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange cannot deliver to queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=276 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=276 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50570 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=276 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50570 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange cannot deliver to queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=278 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=278 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50572 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=278 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50572 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange cannot deliver to queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=280 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=280 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50574 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=280 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50574 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange cannot deliver to queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=282 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=282 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50576 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=282 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50576 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) Traceback (most recent call last):
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return self.dispatch(request, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return handler(request, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) value = method(self, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return func(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 1057, in post
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) async_result = repo_controller.queue_sync_with_auto_publish(repo_id, overrides)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/celery/local.py", line 191, in __call__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return self._get_current_object()(*a, **kw)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return super(PulpTask, self).__call__(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 379, in __call__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return self.run(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 712, in queue_sync_with_auto_publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) kwargs=kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 426, in apply_async_with_reservation
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) queue=RESOURCE_MANAGER_QUEUE
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 99, in apply_async
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return super(PulpTask, self).apply_async(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 535, in apply_async
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) **options
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/celery/app/base.py", line 737, in send_task
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) amqp.send_task_message(P, name, message, **options)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 558, in send_task_message
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) **properties
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) exchange_name, declare,
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 527, in _ensured
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) errback and errback(exc, 0)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) self.gen.throw(type, value, traceback)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) sys.exc_info()[2])
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) yield
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) return fun(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) mandatory=mandatory, immediate=immediate,
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1138, in basic_publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) self._put(routing_key, message, exchange, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 484, in _put
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) sender.close()
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "<string>", line 6, in close
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1031, in close
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) self.sync(timeout=timeout)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "<string>", line 6, in sync
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1015, in sync
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) if not self._ewait(lambda: self.acked >= mno, timeout=timeout):
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 928, in _ewait
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) result = self.session._ewait(lambda: self.error or predicate(), timeout)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) self.check_error()
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) raise e
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) OperationalError: illegal-argument: Value for replyText is too large(320)
Updated by dkliban@redhat.com almost 5 years ago
There is not enough information. I can't reproduce the issue without at least knowing the URL that the sync is happening from.
Updated by dkliban@redhat.com almost 5 years ago
Please disregard the previous comment. Looks like you are having trouble dispatching tasks. There is something going on with Qpid or associated libraries. Most likely an environmental problem.
Updated by ggainey almost 5 years ago
One issue we've seen before is qpid running out of AIO resources. There is a Red Hat article around this error message here : https://access.redhat.com/solutions/1425893
(Note that you will need at least a Developer Sub to RH to read the whole article)
The net is that each consumer wants a durable queue, qpid reserves 33 AIO requests per durable queue, and the default setting for aio-max is 65536. That means that at 1980 consumers, qpid runs out of resources and starts returning errors, which are larger than allowed reply-text-field-size, which causes an exception, which results in the msg you see here.
Try raising fs.aio-max-nr by A Lot. In /etc/sysctl.conf :
fs.aio-max-nr=655360
Then issue sysctl -p , and see if that helps.
Updated by jokabo almost 5 years ago
Currently I have only 7 registered consumers.
I am running pulp in a lxc container with proxmox. Is that a problem?
I've checked the value from the given variable:
Result: fs.aio-max-nr = 1048576
The amout is larger and in a lxc container I was not able to change the value.
Also I was not able to unregister consumers - got an 500 - connection
aborted
[01:01:58 root@repos]{~}>pulp-admin consumer unregister --consumer-id dns1.<name>.intern
An internal error occurred on the Pulp server:
RequestException: DELETE request
on /pulp/api/v2/consumers/repos.<name>.intern/ failed with 500 - connection
aborted
Updated by dkliban@redhat.com almost 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp 2 has only been demonstrated to work inside containers for proof of concept purposes only. We do not have any sort of CI set up to test Pulp 2 running inside containers. So I don't think we will be able to help resolve your issue.
We will be making Pulp 3 containers available soon. However, pulp 3 does not provide any client management functionality.
Updated by amacdona@redhat.com almost 5 years ago
@jokabo, you might try asking on the mailing list https://www.redhat.com/mailman/listinfo/pulp-list
Updated by bmbouter almost 4 years ago
- Category deleted (
14)
We are removing the 'API' category per open floor discussion June 16, 2020.
Actions
.