Project

Profile

Help

Issue #5109

500 - illegal-argument: Value for replyText is too large(320)

Added by jokabo over 2 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Hi,

since some days I got the following error when I try to delete or Sync a Repo with pulp-admin:

[14:13:26 root@repos]{/var/log/mongodb}>pulp-admin rpm repo sync run --force-full --repo-id=jb-centos7-mariadb-main
+----------------------------------------------------------------------+
           Synchronizing Repository [jb-centos7-mariadb-main]
+----------------------------------------------------------------------+

The following sync configuration options will be used:

Force Full:  True

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/jb-centos7-mariadb-main/actions/sync/ failed with
500 - illegal-argument: Value for replyText is too large(320)

I got this error for every project!

Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange  cannot deliver to  queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=276 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=276 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50570 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=276 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50570 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange  cannot deliver to  queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=278 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=278 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50572 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=278 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50572 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange  cannot deliver to  queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=280 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=280 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50574 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=280 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50574 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] warning Exchange  cannot deliver to  queue resource_manager: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=282 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Broker] error Connection exception: framing-error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=282 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50576 closed by error: Queue resource_manager: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=282 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
Jul 10 14:17:16 repos qpidd: 2019-07-10 14:17:16 [Protocol] error Connection qpid.[::1]:5672-[::1]:50576 closed by error: illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) illegal-argument: Value for replyText is too large(320)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) Traceback (most recent call last):
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return self.dispatch(request, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return handler(request, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     value = method(self, *args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return func(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 1057, in post
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     async_result = repo_controller.queue_sync_with_auto_publish(repo_id, overrides)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/celery/local.py", line 191, in __call__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return self._get_current_object()(*a, **kw)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return super(PulpTask, self).__call__(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 379, in __call__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return self.run(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 712, in queue_sync_with_auto_publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     kwargs=kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 426, in apply_async_with_reservation
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     queue=RESOURCE_MANAGER_QUEUE
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 99, in apply_async
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return super(PulpTask, self).apply_async(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 535, in apply_async
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     **options
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/celery/app/base.py", line 737, in send_task
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     amqp.send_task_message(P, name, message, **options)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 558, in send_task_message
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     **properties
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     exchange_name, declare,
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 527, in _ensured
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     errback and errback(exc, 0)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     self.gen.throw(type, value, traceback)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     sys.exc_info()[2])
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     yield
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     return fun(*args, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     mandatory=mandatory, immediate=immediate,
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1138, in basic_publish
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     self._put(routing_key, message, exchange, **kwargs)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 484, in _put
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     sender.close()
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "<string>", line 6, in close
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1031, in close
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     self.sync(timeout=timeout)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "<string>", line 6, in sync
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1015, in sync
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     if not self._ewait(lambda: self.acked >= mno, timeout=timeout):
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 928, in _ewait
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     result = self.session._ewait(lambda: self.error or predicate(), timeout)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     self.check_error()
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592)     raise e
Jul 10 14:17:16 repos pulp: pulp.server.webservices.middleware.exception:ERROR: (308-86592) OperationalError: illegal-argument: Value for replyText is too large(320)

History

#1 Updated by dkliban@redhat.com over 2 years ago

There is not enough information. I can't reproduce the issue without at least knowing the URL that the sync is happening from.

#2 Updated by dkliban@redhat.com over 2 years ago

Please disregard the previous comment. Looks like you are having trouble dispatching tasks. There is something going on with Qpid or associated libraries. Most likely an environmental problem.

#3 Updated by ggainey over 2 years ago

One issue we've seen before is qpid running out of AIO resources. There is a Red Hat article around this error message here : https://access.redhat.com/solutions/1425893

(Note that you will need at least a Developer Sub to RH to read the whole article)

The net is that each consumer wants a durable queue, qpid reserves 33 AIO requests per durable queue, and the default setting for aio-max is 65536. That means that at 1980 consumers, qpid runs out of resources and starts returning errors, which are larger than allowed reply-text-field-size, which causes an exception, which results in the msg you see here.

Try raising fs.aio-max-nr by A Lot. In /etc/sysctl.conf :

fs.aio-max-nr=655360

Then issue sysctl -p , and see if that helps.

#4 Updated by jokabo over 2 years ago

Currently I have only 7 registered consumers.

I am running pulp in a lxc container with proxmox. Is that a problem?

I've checked the value from the given variable:
Result: fs.aio-max-nr = 1048576

The amout is larger and in a lxc container I was not able to change the value.

Also I was not able to unregister consumers - got an 500 - connection
aborted

[01:01:58 root@repos]{~}>pulp-admin consumer unregister --consumer-id dns1.<name>.intern
An internal error occurred on the Pulp server:

RequestException: DELETE request
on /pulp/api/v2/consumers/repos.<name>.intern/ failed with 500 - connection
aborted

#5 Updated by dkliban@redhat.com over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 has only been demonstrated to work inside containers for proof of concept purposes only. We do not have any sort of CI set up to test Pulp 2 running inside containers. So I don't think we will be able to help resolve your issue.

We will be making Pulp 3 containers available soon. However, pulp 3 does not provide any client management functionality.

#6 Updated by amacdona@redhat.com over 2 years ago

@jokabo, you might try asking on the mailing list https://www.redhat.com/mailman/listinfo/pulp-list

#7 Updated by bmbouter over 1 year ago

  • Category deleted (14)

We are removing the 'API' category per open floor discussion June 16, 2020.

Also available in: Atom PDF