Pulp

Thank you for the bug report!

Would you mind including the traceback here instead of in a pastebin? It
helps for others who have the same issue to be able to search and find
this report. Also, pastebins sometimes disappear after time and it's
good to be able to have all the information here for archival purposes.

I didn't see any sensitive information in the traceback, but it would be
a good idea for you to double check as well. Thanks again!

--
Randy Barlow

Thank you Randy,

I am posting traceback from the pastebin here as requested.

Jakub

FAIL: test_13_delete_repo (tests.general_tests.test_02_rpm_repo.SimpleRepoTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/pulp-automation/tests/general_tests/test_02_rpm_repo.py", line 186, in test_13_delete_repo
    Task.wait_for_report(self.pulp, response)
  File "/root/pulp-automation/pulp_auto/task.py", line 149, in wait_for_report
    tasks = Task.from_report(pulp, response)
  File "/root/pulp-automation/pulp_auto/task.py", line 124, in from_report
    assert 'spawned_tasks' in data, 'invalid report data: %s' % data
AssertionError: invalid report data: {u'exception': [u'AssertionFailed: expected topic, got queue\n'], u'traceback': [u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__\n    return self.app(environ, start_response)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__\n    return self.app(environ, start_response)\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 279, in wsgi\n    result = self.handle_with_processors()\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors\n    return process(self.processors)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor\n    return handler()\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor\n    result = handler()\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 28, in process\n    return self.handle()\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle\n    return self._delegate(fn, self.fvars, args)\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 422, in _delegate\n    return f()\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 430, in <lambda>\n    f = lambda: self._delegate_sub_application(pat, what)\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 455, in _delegate_sub_application\n    return app.handle_with_processors()\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors\n    return process(self.processors)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor\n    return handler()\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor\n    result = handler()\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>\n    return p(lambda: process(processors))\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 28, in process\n    return self.handle()\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle\n    return self._delegate(fn, self.fvars, args)\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 420, in _delegate\n    return handle_class(cls)\n', u'  File "/usr/lib/python2.7/site-packages/web/application.py", line 396, in handle_class\n    return tocall(*args)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 203, in _auth_decorator\n    value = method(self, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/repositories.py", line 271, in DELETE\n    [repo_id], tags=task_tags)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 261, in apply_async_with_reservation\n    queue=RESOURCE_MANAGER_QUEUE)\n', u'  File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 555, in apply_async\n    **dict(self._get_exec_options(), **options)\n', u'  File "/usr/lib/python2.7/site-packages/celery/app/base.py", line 351, in send_task\n    reply_to=reply_to or self.oid, **options\n', u'  File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task\n    **kwargs\n', u'  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 168, in publish\n    routing_key, mandatory, immediate, exchange, declare)\n', u'  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured\n    return fun(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 184, in _publish\n    mandatory=mandatory, immediate=immediate,\n', u'  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1094, in basic_publish\n    self._put(routing_key, message, exchange, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 470, in _put\n    sender = self.transport.session.sender(address)\n', u'  File "<string>", line 6, in sender\n', u'  File "/usr/lib/python2.7/site-packages/qpid_python-0.26-py2.7.egg/qpid/messaging/endpoints.py", line 609, in sender\n    raise e\n'], u'_href': u'/pulp/api/v2/repositories/SimpleRepoTest_repo/', u'error_message': u'expected topic, got queue', u'http_request_method': u'DELETE', u'http_status': 500}
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Resetting dropped connection: ec2-54-91-109-123.compute-1.amazonaws.com
py.warnings: WARNING: /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning

requests.packages.urllib3.connectionpool: DEBUG: "DELETE /pulp/api/v2/repositories/SimpleRepoTest_repo/ HTTP/1.1" 500 5665
pulp_auto.pulp: DEBUG: <pulp_auto.pulp.Pulp object at 0x7f8675db6310>.send(Request('DELETE', u'/repositories/SimpleRepoTest_repo/', data='{}', headers={'content-type': 'application/json'})) == <Response [500]>
--------------------- >> end captured logging << ---------------------

I investigated the python-qpid rpm itself to see if the fix for JIRA 5557 was present, and it is, so I don't think that is the problem here. I also tried to reproduce this on a fresh f21 box, but I could not. Below are the steps that I took to reproduce test it. jkvita how are these steps similar or different from your system. Also I couldn't see any difference in between the package versions and the ones that you received. Maybe try this on a fresh 21 installation? If everything does look the same and you still experience the issue I'll suggest manually deleting all of your qpid queues.

0. Install the Pulp GPG key

1. Add the repo file for pulp 2.6 beta. Created /etc/yum.repos.d/pulp-beta.repo with the following contents:

[pulp-beta]
name=pulp-beta
baseurl=https://repos.fedorapeople.org/repos/pulp/pulp/beta/2.6/fedora-21/x86_64/
enabled=1
gpgcheck=1

2. Install all necessary packages

sudo yum install mongodb-server qpid-cpp-server qpid-cpp-server-store
sudo yum groupinstall pulp-server-qpid pulp-admin pulp-consumer-qpid

3. Start mongo and Qpid

sudo service mongod start
sudo service qpidd start

4. Run pulp-manage-db

sudo -u apache pulp-manage-db

5. Open two terminals. One to watch the logs, and the other to start the following services by. With each one, verify that the service starts correctly.

sudo service httpd start
sudo service pulp_celerybeat start
sudo service pulp_workers start
sudo service pulp_resource_manager start

6. I temporarily disable SSL and set the expected hostname for pulp-admin for testing purposes. (not safe for production!) I add these contents to ~/.pulp/admin.conf

[server]
host = <FQDN hostname goes here>
verify_ssl = false

7. Configure pulp-admin

pulp-admin login -u admin -p admin

8. Then I test Pulp which all goes as expected including the repo delete.

pulp-admin -u admin -p admin rpm repo create --repo-id zoo --relative-url zoo --feed http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo
pulp-admin -u admin -p admin rpm repo delete --repo-id zoo

We are not able to reproduce it again either. We have several bugged machines, but new instances are fine and everything passes...
I don't really know what we were doing wrong.