2018-05-30T15:42:13.751112+00:00 pulp1.infra-ng.dev pulp[845]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running. 2018-05-30T15:42:18.032150+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017 2018-05-30T15:42:18.053284+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017 2018-05-30T15:42:18.555117+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {} 2018-05-30T15:42:18.575977+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: deb_release = pulp_deb.plugins.db.models:DebRelease 2018-05-30T15:42:18.602676+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: deb = pulp_deb.plugins.db.models:DebPackage 2018-05-30T15:42:18.603006+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: deb_component = pulp_deb.plugins.db.models:DebComponent 2018-05-30T15:42:18.603286+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata 2018-05-30T15:42:18.742692+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution 2018-05-30T15:42:18.743091+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM 2018-05-30T15:42:18.743401+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup 2018-05-30T15:42:18.743714+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory 2018-05-30T15:42:18.744011+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: iso = pulp_rpm.plugins.db.models:ISO 2018-05-30T15:42:18.744330+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment 2018-05-30T15:42:18.744594+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM 2018-05-30T15:42:18.744863+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks 2018-05-30T15:42:18.745127+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM 2018-05-30T15:42:18.745355+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile 2018-05-30T15:42:18.745670+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: python_package = pulp_python.plugins.models:Package 2018-05-30T15:42:18.765044+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist 2018-05-30T15:42:18.777068+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin deb_distributor for types: deb,deb_component,deb_release 2018-05-30T15:42:18.824502+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file 2018-05-30T15:42:18.825446+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group 2018-05-30T15:42:18.830221+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin rpm_rsync_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file 2018-05-30T15:42:18.834049+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin iso_distributor for types: iso 2018-05-30T15:42:18.835519+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin iso_rsync_distributor for types: iso 2018-05-30T15:42:18.837753+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin python_distributor for types: python_package 2018-05-30T15:42:18.838929+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group 2018-05-30T15:42:18.842562+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin deb_importer for types: deb,deb_component,deb_release 2018-05-30T15:42:18.855203+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment,package_langpacks 2018-05-30T15:42:18.858213+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin iso_importer for types: iso 2018-05-30T15:42:18.860382+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin python_importer for types: python_package 2018-05-30T15:42:18.862025+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_profiler for types: rpm,erratum 2018-05-30T15:42:18.863627+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum for types: rpm 2018-05-30T15:42:18.864460+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin rhui for types: rpm 2018-05-30T15:42:19.013060+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.async.tasks:INFO: Cleaning up shutdown worker 'resource_manager@pulp.ciscolabs.infra-ng.dev'. 2018-05-30T15:42:19.092852+00:00 pulp1.infra-ng.dev pulp[21597]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017 2018-05-30T15:42:19.100393+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.connection:INFO: Connected to amqp://guest:**@127.0.0.1:5672// 2018-05-30T15:42:19.114409+00:00 pulp1.infra-ng.dev pulp[21597]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017 2018-05-30T15:42:19.125764+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.mingle:INFO: mingle: searching for neighbors 2018-05-30T15:42:19.173387+00:00 pulp1.infra-ng.dev pulp[757]: celery.worker.control:INFO: sync with resource_manager@pulp.ciscolabs.infra-ng.dev 2018-05-30T15:42:19.173481+00:00 pulp1.infra-ng.dev pulp[754]: celery.worker.control:INFO: sync with resource_manager@pulp.ciscolabs.infra-ng.dev 2018-05-30T15:42:19.616065+00:00 pulp1.infra-ng.dev pulp[21597]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {} 2018-05-30T15:42:20.185295+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.mingle:INFO: mingle: sync with 2 nodes 2018-05-30T15:42:20.185450+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.mingle:INFO: mingle: sync complete 2018-05-30T15:42:20.204601+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[4d26c382-2f2f-417e-a09f-05097c32a971] 2018-05-30T15:42:20.210805+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) Unrecoverable error: TypeError("'NoneType' object is not callable",) 2018-05-30T15:42:20.210864+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) Traceback (most recent call last): 2018-05-30T15:42:20.210915+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start 2018-05-30T15:42:20.210981+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self.blueprint.start(self) 2018-05-30T15:42:20.211034+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start 2018-05-30T15:42:20.211083+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) step.start(parent) 2018-05-30T15:42:20.211180+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 370, in start 2018-05-30T15:42:20.211260+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self.obj.start() 2018-05-30T15:42:20.211317+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start 2018-05-30T15:42:20.211369+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) blueprint.start(self) 2018-05-30T15:42:20.211421+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start 2018-05-30T15:42:20.211472+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) step.start(parent) 2018-05-30T15:42:20.211524+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 594, in start 2018-05-30T15:42:20.211589+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) c.loop(*c.loop_args()) 2018-05-30T15:42:20.211639+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 47, in asynloop 2018-05-30T15:42:20.211700+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) consumer.consume() 2018-05-30T15:42:20.211758+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 476, in consume 2018-05-30T15:42:20.211808+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self._basic_consume(T, no_ack=no_ack, nowait=False) 2018-05-30T15:42:20.211861+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 597, in _basic_consume 2018-05-30T15:42:20.211909+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) no_ack=no_ack, nowait=nowait) 2018-05-30T15:42:20.212001+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 737, in consume 2018-05-30T15:42:20.212069+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) arguments=self.consumer_arguments) 2018-05-30T15:42:20.212116+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1564, in basic_consume 2018-05-30T15:42:20.212162+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) wait=None if nowait else spec.Basic.ConsumeOk, 2018-05-30T15:42:20.212208+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 59, in send_method 2018-05-30T15:42:20.212254+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self.wait(wait, returns_tuple=returns_tuple) 2018-05-30T15:42:20.212313+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 79, in wait 2018-05-30T15:42:20.212382+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self.connection.drain_events(timeout=timeout) 2018-05-30T15:42:20.212432+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 471, in drain_events 2018-05-30T15:42:20.212480+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) while not self.blocking_read(timeout): 2018-05-30T15:42:20.212529+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 477, in blocking_read 2018-05-30T15:42:20.212608+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self.on_inbound_frame(frame) 2018-05-30T15:42:20.212686+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 77, in on_frame 2018-05-30T15:42:20.212742+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) callback(channel, msg.frame_method, msg.frame_args, msg) 2018-05-30T15:42:20.212793+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 481, in on_inbound_method 2018-05-30T15:42:20.212890+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) method_sig, payload, content, 2018-05-30T15:42:20.212943+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 128, in dispatch_method 2018-05-30T15:42:20.213012+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) listener(*args) 2018-05-30T15:42:20.213064+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1599, in _on_basic_deliver 2018-05-30T15:42:20.213114+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) fun(msg) 2018-05-30T15:42:20.213174+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 623, in _receive_callback 2018-05-30T15:42:20.213230+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return on_m(message) if on_m else self.receive(decoded, message) 2018-05-30T15:42:20.213283+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 568, in on_task_received 2018-05-30T15:42:20.213349+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) callbacks, 2018-05-30T15:42:20.213399+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/strategy.py", line 145, in task_message_handler 2018-05-30T15:42:20.213449+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) handle(req) 2018-05-30T15:42:20.213498+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 221, in _process_task_sem 2018-05-30T15:42:20.213598+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self._quick_acquire(self._process_task, req) 2018-05-30T15:42:20.213679+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/async/semaphore.py", line 62, in acquire 2018-05-30T15:42:20.213742+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) callback(*partial_args, **partial_kwargs) 2018-05-30T15:42:20.213789+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 226, in _process_task 2018-05-30T15:42:20.213849+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) req.execute_using_pool(self.pool) 2018-05-30T15:42:20.213896+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/request.py", line 531, in execute_using_pool 2018-05-30T15:42:20.213942+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) correlation_id=task_id, 2018-05-30T15:42:20.214041+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/concurrency/base.py", line 155, in apply_async 2018-05-30T15:42:20.214088+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) **options) 2018-05-30T15:42:20.214133+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1486, in apply_async 2018-05-30T15:42:20.214178+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self._quick_put((TASK, (result._job, None, func, args, kwds))) 2018-05-30T15:42:20.214226+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) TypeError: 'NoneType' object is not callable 2018-05-30T15:42:21.734082+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.async.tasks:INFO: Cleaning up shutdown worker 'resource_manager@pulp.ciscolabs.infra-ng.dev'. 2018-05-30T15:42:25.770581+00:00 pulp1.infra-ng.dev pulp[845]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.