Actions
Issue #1342
closed
Automation tests failures on rhel 7.1
Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.7.0 Beta
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
Many tests fails in automation on rhel 7.1 with output shown below, but everything passes on rhel 7.2 beta. Reproduced manually on binding importer to a repository:
$ curl -X POST -d '{"importer_type_id": "yum_importer", "importer_config": {"feed": "https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/", "ssl_validation": false}, "id": "yum_importer"}' https://servername/pulp/api/v2/repositories/
zoo/importers/ -u admin:admin -k -vvvv
* About to connect() to servername.com port 443 (#0)
* Trying 10.9.4.54...
* Connected to servername.com (10.9.4.54) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* subject: CN=servername,O=Ltd,ST=NC,C=US
* start date: Oct 29 15:50:58 2015 GMT
* expire date: Oct 28 15:50:58 2016 GMT
* common name: servername.com
* issuer: CN=servername.com,O=Ltd,L=Raleigh,ST=NC,C=US
* Server auth using Basic with user 'admin'
> POST /pulp/api/v2/repositories/zoo/importers/ HTTP/1.1
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.29.0
> Host: servername.com
> Accept: */*
> Content-Length: 178
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 178 out of 178 bytes
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* skipping SSL peer certificate verification
< HTTP/1.1 500 INTERNAL SERVER ERROR
< Date: Thu, 29 Oct 2015 17:32:02 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
< Content-Length: 4122
< Connection: close
< Content-Type: application/json; charset=utf-8
<
{"http_request_method": "POST", "exception": ["Exception: Exception from Agent: {u'error_code': 7, u'error_text': 'Queue resource_manager: create() failed:
jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file=\"/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use
/4b957fed-d3f3-4bf1-9766-562ec80fc1eb.jrnl\" symlink=\"/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/4b957fed-d3f3-4bf1-9766-562ec80fc1eb.jrnl\" errno=13
(Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)'}\n"], "error_message": "Exception from Agent:
{u'error_code': 7, u'error_text': 'Queue resource_manager: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link
operation failed (file=\"/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/4b957fed-d3f3-4bf1-9766-562ec80fc1eb.jrnl\" symlink=\"/var/lib/qpidd/.qpidd/qls/jrnl2
/resource_manager/4b957fed-d3f3-4bf1-9766-562ec80fc1eb.jrnl\" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore
/MessageStoreImpl.cpp:425)'}", "_href": "/pulp/api/v2/repositories/zoo/importers/", "http_status": 500, "traceback": [" File \"/usr/lib/python2.7/site-packages
/django/core/handlers/base.py\", line 112, in get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", " File \"/usr
/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n return self.dispatch(request, *args, **kwargs)\n", " File \"/usr
/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n return handler(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7
/site-packages/pulp/server/webservices/views/decorators.py\", line 237, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method,
*args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 191, in _verify_auth\n value =
method(self, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 111, in wrapper\n return
func(*args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 448, in post\n
{'repo_plugin_config': config}, tags=task_tags)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 323, in
apply_async_with_reservation\n queue=RESOURCE_MANAGER_QUEUE)\n", " File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 555, in
apply_async\n **dict(self._get_exec_options(), **options)\n", " File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 351, in send_task\n
reply_to=reply_to or self.oid, **options\n", " File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 305, in publish_task\n **kwargs\n", "
File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 168, in publish\n routing_key, mandatory, immediate, exchange, declare)\n", " File \"/usr
/lib/python2.7/site-packages/kombu/connection.py\", line 436, in _ensured\n return fun(*args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/kombu
/messaging.py\", line 180, in _publish\n [maybe_declare(entity) for entity in declare]\n", " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\",
line 111, in maybe_declare\n return maybe_declare(entity, self.channel, retry, **retry_policy)\n", " File \"/usr/lib/python2.7/site-packages/kombu
/common.py\", line 113, in maybe_declare\n return _maybe_declare(entity, declared, ident, channel)\n", " File \"/usr/lib/python2.7/site-packages/kombu
/common.py\", line 120, in _maybe_declare\n entity.declare()\n", " File \"/usr/lib/python2.7/site-packages/kombu/entity.py\", line 505, in declare\n
self.queue_declare(nowait, passive=False)\n", " File \"/usr/lib/python2.7/site-packages/kombu/entity.py\", line 531, in queue_declare\n nowait=nowait)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 696, in queue_d* Closing connection 0
Updated by mhrivnak over 8 years ago
Please add the versions of qpid packages in use.
Also, were there any log messages about qpid failures or errors? Also consult audit.log please.
Updated by placko over 8 years ago
$ yum list installed |grep qpid
python-gofer-qpid.noarch 2.6.6-2.el7 @pulp-2.7-beta
python-qpid.noarch 0.32-9.el7 @epel
python-qpid-common.noarch 0.32-9.el7 @epel
python-qpid-qmf.x86_64 0.32-1.el7 @epel
qpid-cpp-client.x86_64 0.34-4.el7 @epel
qpid-cpp-server.x86_64 0.34-4.el7 @epel
qpid-cpp-server-linearstore.x86_64
qpid-proton-c.x86_64 0.10-2.el7 @epel
qpid-qmf.x86_64 0.32-1.el7 @epel
qpid-tools.noarch 0.32-9.el7 @epel
pulp_resource_manager service keeps disabling, so after starting and making request to server, journalctl contains:
-- Logs begin at Thu 2015-10-29 11:38:05 EDT, end at Fri 2015-10-30 11:31:21 EDT. --
Oct 30 11:26:19 pulp-server.com systemd[1]: Starting Pulp Resource Manager...
Oct 30 11:26:19 pulp-server.com systemd[1]: Started Pulp Resource Manager.
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) pulp.task
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) Traceback (most recent call last):
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._reader.open()
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) return fn(*args, **keywords)
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._impl.open()
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) raise NotFound(*e.args)
Oct 30 11:26:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) NotFound: no such queue: pulp.task
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) /usr/lib/python2.7/site-packages/celery/apps/worker.py:161: CDeprecationWarning:
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) Starting from version 3.2 Celery will refuse to accept pickle by default.
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) The pickle serializer is a security concern as it may give attackers
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) the ability to execute any command. It's important to secure
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) your broker from unauthorized access when using pickle, so we think
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) that enabling pickle should require a deliberate action and not be
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) the default choice.
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) If you depend on pickle then you should set a setting to disable this
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) warning and to be sure that everything will continue working
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) when you upgrade to Celery 3.2::
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) You must only enable the serializers that you will actually use.
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352) warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
Oct 30 11:26:20 pulp-server.com pulp[3592]: py.warnings:WARNING: (3592-84352)
Oct 30 11:26:20 pulp-server.com celery[3592]: -------------- resource_manager@pulp-server.com v3.1.11 (Cipater)
Oct 30 11:26:20 pulp-server.com celery[3592]: ---- **** -----
Oct 30 11:26:20 pulp-server.com celery[3592]: --- * *** * -- Linux-3.10.0-229.el7.x86_64-x86_64-with-redhat-7.1-Maipo
Oct 30 11:26:20 pulp-server.com celery[3592]: -- * - **** ---
Oct 30 11:26:20 pulp-server.com celery[3592]: - ** ---------- [config]
Oct 30 11:26:20 pulp-server.com celery[3592]: - ** ---------- .> app: tasks:0x2c3dc50
Oct 30 11:26:20 pulp-server.com celery[3592]: - ** ---------- .> transport: qpid://pulp-server.com:5672//
Oct 30 11:26:20 pulp-server.com celery[3592]: - ** ---------- .> results: disabled
Oct 30 11:26:20 pulp-server.com celery[3592]: - *** --- * --- .> concurrency: 1 (prefork)
Oct 30 11:26:20 pulp-server.com celery[3592]: -- ******* ----
Oct 30 11:26:20 pulp-server.com celery[3592]: --- ***** ----- [queues]
Oct 30 11:26:20 pulp-server.com celery[3592]: -------------- .> resource_manager exchange=resource_manager(direct) key=resource_manager
Oct 30 11:26:20 pulp-server.com celery[3592]: .> resource_manager@pulp-server.com.dq exchange=C.dq(direct) key=resource_manager@pulp-server.com
Oct 30 11:26:20 pulp-server.com pulp[3592]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Oct 30 11:26:20 pulp-server.com pulp[3592]: celery.worker.consumer:INFO: Connected to qpid://pulp-server.com:5672//
Oct 30 11:26:20 pulp-server.com pulp[3592]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) Unrecoverable error: Exception('Exception from Agent: {u\'error_code\': 7, u\'error_text\': \'Queue resource_manager: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/834ce58b-5ae7-4c95-938d-bb6d3a09976b.jrnl" symlink="/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/834ce58b-5ae7-4c95-938d-bb6d3a09976b.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)\'}',)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) Traceback (most recent call last):
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) self.blueprint.start(self)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) step.start(parent)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 373, in start
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) return self.obj.start()
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) blueprint.start(self)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) step.start(parent)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 608, in start
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) c.connection, on_decode_error=c.on_decode_error,
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 375, in __init__
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) **kw
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 359, in __init__
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) self.revive(self.channel)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 371, in revive
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) self.declare()
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 381, in declare
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) queue.declare()
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 505, in declare
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) self.queue_declare(nowait, passive=False)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 531, in queue_declare
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) nowait=nowait)
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 696, in queue_declare
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) raise exc
Oct 30 11:26:21 pulp-server.com pulp[3592]: celery.worker:ERROR: (3592-84352) Exception: Exception from Agent: {u'error_code': 7, u'error_text': 'Queue resource_manager: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/834ce58b-5ae7-4c95-938d-bb6d3a09976b.jrnl" symlink="/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/834ce58b-5ae7-4c95-938d-bb6d3a09976b.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)'}
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) pulp.task
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) Traceback (most recent call last):
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._reader.open()
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) return fn(*args, **keywords)
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._impl.open()
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) raise NotFound(*e.args)
Oct 30 11:26:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) NotFound: no such queue: pulp.task
Oct 30 11:26:26 pulp-server.com qpidd[11251]: 2015-10-30 11:26:26 [Management] error Detected two management objects with the same identifier: 0-0-1--4519(org.apache.qpid.broker:queue:resource_manager)
Oct 30 11:26:33 pulp-server.com sshd[3564]: Received disconnect from 213.175.37.10: 11: disconnected by user
Oct 30 11:26:33 pulp-server.com sshd[3564]: pam_unix(sshd:session): session closed for user root
Oct 30 11:26:33 pulp-server.com systemd-logind[21809]: Removed session 35.
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) Exception from Agent: {u'error_code': 7, u'error_text': 'Queue resource_manager: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/fce58f36-b3ec-4e9d-9006-fb06cd90ec80.jrnl" symlink="/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/fce58f36-b3ec-4e9d-9006-fb06cd90ec80.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)'}
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) Traceback (most recent call last):
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) response = wrapped_callback(request, *callback_args, **callback_kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return self.dispatch(request, *args, **kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return handler(request, *args, **kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 237, in _auth_decorator
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 191, in _verify_auth
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) value = method(self, *args, **kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 111, in wrapper
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return func(*args, **kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 448, in post
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) {'repo_plugin_config': config}, tags=task_tags)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 323, in apply_async_with_reservation
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) queue=RESOURCE_MANAGER_QUEUE)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 555, in apply_async
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) **dict(self._get_exec_options(), **options)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/celery/app/base.py", line 351, in send_task
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) reply_to=reply_to or self.oid, **options
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) **kwargs
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 168, in publish
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) routing_key, mandatory, immediate, exchange, declare)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return fun(*args, **kwargs)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 180, in _publish
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) [maybe_declare(entity) for entity in declare]
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 111, in maybe_declare
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return maybe_declare(entity, self.channel, retry, **retry_policy)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/common.py", line 113, in maybe_declare
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) return _maybe_declare(entity, declared, ident, channel)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/common.py", line 120, in _maybe_declare
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) entity.declare()
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 505, in declare
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) self.queue_declare(nowait, passive=False)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 531, in queue_declare
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) nowait=nowait)
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 696, in queue_declare
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) raise exc
Oct 30 11:26:35 pulp-server.com pulp[11311]: pulp.server.webservices.middleware.exception:ERROR: (11311-12128) Exception: Exception from Agent: {u'error_code': 7, u'error_text': 'Queue resource_manager: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/fce58f36-b3ec-4e9d-9006-fb06cd90ec80.jrnl" symlink="/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/fce58f36-b3ec-4e9d-9006-fb06cd90ec80.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)'}
Oct 30 11:26:36 pulp-server.com qpidd[11251]: 2015-10-30 11:26:36 [Management] error Detected two management objects with the same identifier: 0-0-1--4522(org.apache.qpid.broker:queue:resource_manager)
Oct 30 11:26:39 pulp-server.com sshd[3613]: Accepted publickey for root from 213.175.37.10 port 4399 ssh2: RSA ca:84:74:77:b6:4c:37:2f:dd:b9:9b:ad:45:f0:47:c9
Oct 30 11:26:39 pulp-server.com systemd[1]: Created slice user-0.slice.
Oct 30 11:26:39 pulp-server.com systemd[1]: Starting Session 36 of user root.
Oct 30 11:26:39 pulp-server.com systemd[1]: Started Session 36 of user root.
Oct 30 11:26:39 pulp-server.com systemd-logind[21809]: New session 36 of user root.
Oct 30 11:26:39 pulp-server.com sshd[3613]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 30 11:26:52 pulp-server.com pulp[11503]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) pulp.task
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) Traceback (most recent call last):
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._reader.open()
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) return fn(*args, **keywords)
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._impl.open()
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) raise NotFound(*e.args)
Oct 30 11:27:18 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) NotFound: no such queue: pulp.task
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) pulp.task
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) Traceback (most recent call last):
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._reader.open()
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) return fn(*args, **keywords)
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._impl.open()
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) raise NotFound(*e.args)
Oct 30 11:27:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) NotFound: no such queue: pulp.task
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) pulp.task
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) Traceback (most recent call last):
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._reader.open()
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) return fn(*args, **keywords)
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._impl.open()
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) raise NotFound(*e.args)
Oct 30 11:27:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) NotFound: no such queue: pulp.task
Oct 30 11:27:52 pulp-server.com pulp[11503]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) pulp.task
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) Traceback (most recent call last):
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._reader.open()
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) return fn(*args, **keywords)
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._impl.open()
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) raise NotFound(*e.args)
Oct 30 11:28:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) NotFound: no such queue: pulp.task
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) pulp.task
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) Traceback (most recent call last):
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._reader.open()
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) return fn(*args, **keywords)
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._impl.open()
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) raise NotFound(*e.args)
Oct 30 11:28:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) NotFound: no such queue: pulp.task
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) pulp.task
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) Traceback (most recent call last):
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._reader.open()
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) return fn(*args, **keywords)
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._impl.open()
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) raise NotFound(*e.args)
Oct 30 11:28:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) NotFound: no such queue: pulp.task
Oct 30 11:28:52 pulp-server.com pulp[11503]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) pulp.task
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) Traceback (most recent call last):
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._reader.open()
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) return fn(*args, **keywords)
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._impl.open()
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) raise NotFound(*e.args)
Oct 30 11:29:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) NotFound: no such queue: pulp.task
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) pulp.task
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) Traceback (most recent call last):
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._reader.open()
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) return fn(*args, **keywords)
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._impl.open()
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) raise NotFound(*e.args)
Oct 30 11:29:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) NotFound: no such queue: pulp.task
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) pulp.task
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) Traceback (most recent call last):
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._reader.open()
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) return fn(*args, **keywords)
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._impl.open()
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) raise NotFound(*e.args)
Oct 30 11:29:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) NotFound: no such queue: pulp.task
Oct 30 11:29:52 pulp-server.com pulp[11503]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) pulp.task
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) Traceback (most recent call last):
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._reader.open()
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) return fn(*args, **keywords)
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._impl.open()
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) raise NotFound(*e.args)
Oct 30 11:30:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) NotFound: no such queue: pulp.task
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) pulp.task
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) Traceback (most recent call last):
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._reader.open()
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) return fn(*args, **keywords)
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._impl.open()
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) raise NotFound(*e.args)
Oct 30 11:30:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) NotFound: no such queue: pulp.task
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) pulp.task
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) Traceback (most recent call last):
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._reader.open()
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) return fn(*args, **keywords)
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._impl.open()
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) raise NotFound(*e.args)
Oct 30 11:30:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) NotFound: no such queue: pulp.task
Oct 30 11:30:52 pulp-server.com pulp[11503]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) pulp.task
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) Traceback (most recent call last):
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._reader.open()
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) return fn(*args, **keywords)
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) self._impl.open()
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) raise NotFound(*e.args)
Oct 30 11:31:19 pulp-server.com pulp[11311]: gofer.messaging.consumer:ERROR: (11311-69120) NotFound: no such queue: pulp.task
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) pulp.task
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) Traceback (most recent call last):
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._reader.open()
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) return fn(*args, **keywords)
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) self._impl.open()
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) raise NotFound(*e.args)
Oct 30 11:31:20 pulp-server.com pulp[11313]: gofer.messaging.consumer:ERROR: (11313-69120) NotFound: no such queue: pulp.task
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) pulp.task
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) Traceback (most recent call last):
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 68, in _open
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._reader.open()
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) return fn(*args, **keywords)
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 587, in open
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) self._impl.open()
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/qpid/reliability.py", line 34, in _fn
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) raise NotFound(*e.args)
Oct 30 11:31:21 pulp-server.com pulp[11312]: gofer.messaging.consumer:ERROR: (11312-69120) NotFound: no such queue: pulp.task
audit.log says nothing (after restarting services, creating repository and making query again)
Updated by bmbouter over 8 years ago
placko, can you try to reproduce this on a different machine with a fresh install? I would like to rule out environmental problems specific to that install.
The original traceback shows errors in qpidd as it tries to store a message in its storage backend (linearstore). If qpidd raises exceptions when you send it messages it would produce these types of errors in Pulp. If you are able to reproduce it on another machine, perhaps those Qpid packages have a problem in them.
You can use qpid.messaging (outside of Pulp) to trigger the same problem which would help isolate Pulp problems from Qpid problems. Use the qpid-tools commands to make a durable=True queue to use for testing. Then use a simple python script to send a message into it. This is basically what Pulp does. Here is an incomplete snippet to get you started:
from qpid.messaging import Connection
connection = Connection(host='localhost')
connection.open()
session = connection.session()
# from the session make a sender and send the message
# https://qpid.apache.org/releases/qpid-0.32/messaging-api/python/api/index.html
session.close()
connection.close()
Note, pulp.task is created when pulp-manage-db runs (IIRC) so the fact that its missing seems like qpid is pretty broken.
If you determine it's an upstream Qpid issue, file a bug with them here: https://issues.apache.org/jira/browse/
I'm interested in this issue so when you have more information post it here please.
Updated by placko over 8 years ago
I reproduced it in clean environment and looks like qpid bug, since qpid fails on following command with same Exception:
qpid-config add queue --durable "queue"
Queue is however created succesfully without --durable switch.
I already filled an issue [1].
Updated by placko over 8 years ago
Problem was in selinux policy anyway, fixed in latest selinux-policy-3.13.1-23 package which is already in upstream.
Updated by placko over 8 years ago
- Status changed from NEW to CLOSED - CURRENTRELEASE
Updated by bmbouter over 8 years ago
- Status changed from CLOSED - CURRENTRELEASE to CLOSED - NOTABUG
- Triaged changed from No to Yes
placko wrote:
Problem was in selinux policy anyway, fixed in latest selinux-policy-3.13.1-23 package which is already in upstream.
placko, thanks for resolving this!
To clarify the above comment 5, the selinux-policy-3.13.1-23 is currently available in the RHEL 7.1 repo. Doing a yum update will resolve this problem for a user.
Updated by bmbouter almost 4 years ago
- Category deleted (
14)
We are removing the 'API' category per open floor discussion June 16, 2020.
Actions
.