Project

Profile

Help

Issue #525

"PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'reserved_resource_worker-0@fqdn.dq' in vhost '/' not equivalent" when using RabbitMQ

Added by neoice@neoice.net over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.4.0
Platform Release:
2.6.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem:

when submitting jobs via `pulp-admin`, application 500s with "Unhandled Exception". syslog shows:

pulp: pulp.server.webservices.middleware.exception:ERROR: PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue '' in vhost '/' not equivalent

Version-Release number of selected component (if applicable):

Name : pulp-server
Arch : noarch
Version : 2.4.0
Release : 1.el6
Size : 2.7 M

Steps to Reproduce:
1. using the Puppet module https://forge.puppetlabs.com/jcline/pulp:

class { '::pulp::server':
  default_login    => 'admin',
  default_password => 'admin',
  msg_transport    => 'amqplib',
  msg_url          => 'tcp://127.0.0.1:5672',
  tasks_broker_url => 'amqp://127.0.0.1:5672',
  require          => [
                        Package['python-qpid-qmf'],
                        Package['python-amqplib'],
                        Package['python-gofer-amqplib'],
                        Service['mongod'],
                        Class['profiles::rabbitmq::pulp'],
                      ],  
}

(there are some additional parts not included)

2. this results in the following being added to /etc/pulp/server.conf:

[messaging]
url: tcp://127.0.0.1:5672
transport: amqplib

[tasks]
broker_url: amqp://127.0.0.1:5672

3. restart all Pulp services

4. submit a job via `pulp-admin`:

pulp-admin rpm repo sync run --repo-id jenkins

Actual results:

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/jenkins/actions/sync/ failed with 500 - Unhandled
Exception

Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in call
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in call
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.handle_with_processors()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
1 pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return f()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: f = lambda: self._delegate_sub_application(pat, what)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return app.handle_with_processors()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_procesors
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return handle_class(cls)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return tocall(*args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: value = method(self, *args, **kwargs)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/repositories.py", line 710, in POST
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: repo_id, [repo_id, overrides], {}, tags=task_tags)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 240, in apply_async_with_reservation
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: _queue_release_resource.apply_async((resource_id,), queue=queue)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 555, in apply_async
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: **dict(self._get_exec_options(), **options)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/base.py", line 351, in send_task
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: reply_to=reply_to or self.oid, **options
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/amqp.py", line 305, in publish_task
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: **kwargs
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 168, in publish
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: routing_key, mandatory, immediate, exchange, declare)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 440, in _ensured
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return fun(*args, **kwargs)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 180, in _publish
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: [maybe_declare(entity) for entity in declare]
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 111, in maybe_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return maybe_declare(entity, self.channel, retry, **retry_policy)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/common.py", line 99, in maybe_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return _maybe_declare(entity)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/common.py", line 110, in _maybe_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: entity.declare()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 505, in declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: self.queue_declare(nowait, passive=False)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 531, in queue_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: nowait=nowait)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 1258, in queue_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: (50, 11), # Channel.queue_declare_ok
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 69, in wait
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return self.dispatch_method(method_sig, args, content)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: return amqp_method(self, args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 241, in _close
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: reply_code, reply_text, (class_id, method_id), ChannelError,
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue '' in vhost '/' not equivalent

Expected results:

queue should be created properly and job should be submitted and run without error.

Additional info:

using Puppet+TheForeman, I can rebuild my host fully from kickstart to this error state. I intend to test this bug by destroying my Pulp VM and recreating it until it works :)

+ This bug was cloned from Bugzilla Bug #1141336 +

History

#1 Updated by rbarlow over 6 years ago

Brian and I spent some time trying to figure this out yesterday, and we believe it is a bug in Celery[0].

The good news is that we found a way to work around the issue, and we've included that fix in PR #1158[1].

Thanks to Brian for taking the time to sort this out with me for the price of a can of La Croix!

[0] https://github.com/celery/celery/issues/2269
[1] https://github.com/pulp/pulp/pull/1158

+ This comment was cloned from Bugzilla #1141336 comment 1 +

#2 Updated by bmbouter over 6 years ago

Merged to 2.5-dev, master

+ This comment was cloned from Bugzilla #1141336 comment 2 +

#3 Updated by cduryee over 6 years ago

fixed in pulp 2.6.0-0.2.beta

+ This comment was cloned from Bugzilla #1141336 comment 3 +

#4 Updated by igulina@redhat.com over 6 years ago

  • Status changed from 5 to 6

[messaging]
#url: tcp://localhost:5672
url: tcp://bunny:bunny_pass@127.0.0.1:5672/pulp

  1. transport: qpid
    transport: amqp

[tasks]

  1. broker_url: qpid://guest@localhost/
    broker_url: amqp://bunny:bunny_pass@127.0.0.1:5672/pulp

rpm -qa amqp

python-amqp-1.4.6-1.el6.noarch
python-amqplib-0.6.1-2.el6.noarch
python-gofer-amqp-2.5.3-1.el6.noarch

rpm -qa pulp-server

pulp-server-2.6.0-0.7.beta.el6.noarch

pulp-admin rpm repo create --repo-id bobik --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/

Successfully created repository [bobik]

pulp-admin rpm repo sync run --repo-id bobik

--------------------------------------------------------------------
Synchronizing Repository [bobik]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

pulp-admin repo list

--------------------------------------------------------------------
Repositories
--------------------------------------------------------------------

Id: myrepo2
Display Name: myrepo2
Description: None
Content Unit Counts:

Id: bobik
Display Name: bobik
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Rpm: 32

less /var/log/messages

Mar 6 12:56:13 XXX pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tas
ks._queue_reserved_task[ba5cfdce-e8cb-4ad7-a2db-8f21e696f53c]
Mar 6 12:56:13 XXX pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[
80743f84-d82a-4667-bc81-f062a6687004] succeeded in 5.78948656213s: <pulp.server.async.tasks.TaskResult o
bject at 0x4493550>
Mar 6 12:56:13 XXX pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_res
ource[e63644f9-9213-43b3-90d5-292bd425c9e7] succeeded in 0.0291958639864s: None
Mar 6 12:56:13 XXX pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.
repo.publish.publish[2a745902-1b26-4952-992b-6ecd5622dea1]
Mar 6 12:56:13 XXX pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tas
ks._release_resource[2bd97358-9bac-477f-b482-f2ce778e9fdc]
Mar 6 12:56:13 XXX pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reser
ved_task[ba5cfdce-e8cb-4ad7-a2db-8f21e696f53c] succeeded in 0.0397787909023s: None
Mar 6 12:56:13 XXX pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.pu
blish[2a745902-1b26-4952-992b-6ecd5622dea1] succeeded in 0.250163258053s: {'exception': None, 'repo_id':
u'bobik', 'traceback': None, 'started': '2015-03-06T17:56:13Z', '_ns':...
Mar 6 12:56:13 XXX pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[2bd97358-9bac-477f-b482-f2ce778e9fdc] succeeded in 0.0117468929384s: None

#5 Updated by bmbouter over 6 years ago

  • Severity changed from High to 3. High

#6 Updated by rbarlow over 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#8 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF