Project

Profile

Help

Issue #3709 ยป pulp-log.txt

pulp_resource_manger start error - signed8bit, 05/30/2018 05:47 PM

 
2018-05-30T15:42:13.751112+00:00 pulp1.infra-ng.dev pulp[845]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
2018-05-30T15:42:18.032150+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
2018-05-30T15:42:18.053284+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
2018-05-30T15:42:18.555117+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
2018-05-30T15:42:18.575977+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: deb_release = pulp_deb.plugins.db.models:DebRelease
2018-05-30T15:42:18.602676+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: deb = pulp_deb.plugins.db.models:DebPackage
2018-05-30T15:42:18.603006+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: deb_component = pulp_deb.plugins.db.models:DebComponent
2018-05-30T15:42:18.603286+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
2018-05-30T15:42:18.742692+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
2018-05-30T15:42:18.743091+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
2018-05-30T15:42:18.743401+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
2018-05-30T15:42:18.743714+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
2018-05-30T15:42:18.744011+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
2018-05-30T15:42:18.744330+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
2018-05-30T15:42:18.744594+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
2018-05-30T15:42:18.744863+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
2018-05-30T15:42:18.745127+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
2018-05-30T15:42:18.745355+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
2018-05-30T15:42:18.745670+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading unit model: python_package = pulp_python.plugins.models:Package
2018-05-30T15:42:18.765044+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist
2018-05-30T15:42:18.777068+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin deb_distributor for types: deb,deb_component,deb_release
2018-05-30T15:42:18.824502+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
2018-05-30T15:42:18.825446+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
2018-05-30T15:42:18.830221+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin rpm_rsync_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
2018-05-30T15:42:18.834049+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin iso_distributor for types: iso
2018-05-30T15:42:18.835519+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin iso_rsync_distributor for types: iso
2018-05-30T15:42:18.837753+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin python_distributor for types: python_package
2018-05-30T15:42:18.838929+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
2018-05-30T15:42:18.842562+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin deb_importer for types: deb,deb_component,deb_release
2018-05-30T15:42:18.855203+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment,package_langpacks
2018-05-30T15:42:18.858213+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin iso_importer for types: iso
2018-05-30T15:42:18.860382+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin python_importer for types: python_package
2018-05-30T15:42:18.862025+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_profiler for types: rpm,erratum
2018-05-30T15:42:18.863627+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin yum for types: rpm
2018-05-30T15:42:18.864460+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.plugins.loader.manager:INFO: Loaded plugin rhui for types: rpm
2018-05-30T15:42:19.013060+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.async.tasks:INFO: Cleaning up shutdown worker 'resource_manager@pulp.ciscolabs.infra-ng.dev'.
2018-05-30T15:42:19.092852+00:00 pulp1.infra-ng.dev pulp[21597]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
2018-05-30T15:42:19.100393+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.connection:INFO: Connected to amqp://guest:**@127.0.0.1:5672//
2018-05-30T15:42:19.114409+00:00 pulp1.infra-ng.dev pulp[21597]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
2018-05-30T15:42:19.125764+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.mingle:INFO: mingle: searching for neighbors
2018-05-30T15:42:19.173387+00:00 pulp1.infra-ng.dev pulp[757]: celery.worker.control:INFO: sync with resource_manager@pulp.ciscolabs.infra-ng.dev
2018-05-30T15:42:19.173481+00:00 pulp1.infra-ng.dev pulp[754]: celery.worker.control:INFO: sync with resource_manager@pulp.ciscolabs.infra-ng.dev
2018-05-30T15:42:19.616065+00:00 pulp1.infra-ng.dev pulp[21597]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
2018-05-30T15:42:20.185295+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.mingle:INFO: mingle: sync with 2 nodes
2018-05-30T15:42:20.185450+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.consumer.mingle:INFO: mingle: sync complete
2018-05-30T15:42:20.204601+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[4d26c382-2f2f-417e-a09f-05097c32a971]
2018-05-30T15:42:20.210805+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) Unrecoverable error: TypeError("'NoneType' object is not callable",)
2018-05-30T15:42:20.210864+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) Traceback (most recent call last):
2018-05-30T15:42:20.210915+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start
2018-05-30T15:42:20.210981+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self.blueprint.start(self)
2018-05-30T15:42:20.211034+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
2018-05-30T15:42:20.211083+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) step.start(parent)
2018-05-30T15:42:20.211180+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 370, in start
2018-05-30T15:42:20.211260+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self.obj.start()
2018-05-30T15:42:20.211317+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
2018-05-30T15:42:20.211369+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) blueprint.start(self)
2018-05-30T15:42:20.211421+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
2018-05-30T15:42:20.211472+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) step.start(parent)
2018-05-30T15:42:20.211524+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 594, in start
2018-05-30T15:42:20.211589+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) c.loop(*c.loop_args())
2018-05-30T15:42:20.211639+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 47, in asynloop
2018-05-30T15:42:20.211700+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) consumer.consume()
2018-05-30T15:42:20.211758+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 476, in consume
2018-05-30T15:42:20.211808+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self._basic_consume(T, no_ack=no_ack, nowait=False)
2018-05-30T15:42:20.211861+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 597, in _basic_consume
2018-05-30T15:42:20.211909+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) no_ack=no_ack, nowait=nowait)
2018-05-30T15:42:20.212001+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 737, in consume
2018-05-30T15:42:20.212069+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) arguments=self.consumer_arguments)
2018-05-30T15:42:20.212116+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1564, in basic_consume
2018-05-30T15:42:20.212162+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) wait=None if nowait else spec.Basic.ConsumeOk,
2018-05-30T15:42:20.212208+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 59, in send_method
2018-05-30T15:42:20.212254+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self.wait(wait, returns_tuple=returns_tuple)
2018-05-30T15:42:20.212313+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 79, in wait
2018-05-30T15:42:20.212382+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self.connection.drain_events(timeout=timeout)
2018-05-30T15:42:20.212432+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 471, in drain_events
2018-05-30T15:42:20.212480+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) while not self.blocking_read(timeout):
2018-05-30T15:42:20.212529+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 477, in blocking_read
2018-05-30T15:42:20.212608+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self.on_inbound_frame(frame)
2018-05-30T15:42:20.212686+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 77, in on_frame
2018-05-30T15:42:20.212742+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) callback(channel, msg.frame_method, msg.frame_args, msg)
2018-05-30T15:42:20.212793+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 481, in on_inbound_method
2018-05-30T15:42:20.212890+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) method_sig, payload, content,
2018-05-30T15:42:20.212943+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 128, in dispatch_method
2018-05-30T15:42:20.213012+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) listener(*args)
2018-05-30T15:42:20.213064+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1599, in _on_basic_deliver
2018-05-30T15:42:20.213114+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) fun(msg)
2018-05-30T15:42:20.213174+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 623, in _receive_callback
2018-05-30T15:42:20.213230+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return on_m(message) if on_m else self.receive(decoded, message)
2018-05-30T15:42:20.213283+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 568, in on_task_received
2018-05-30T15:42:20.213349+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) callbacks,
2018-05-30T15:42:20.213399+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/strategy.py", line 145, in task_message_handler
2018-05-30T15:42:20.213449+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) handle(req)
2018-05-30T15:42:20.213498+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 221, in _process_task_sem
2018-05-30T15:42:20.213598+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) return self._quick_acquire(self._process_task, req)
2018-05-30T15:42:20.213679+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/kombu/async/semaphore.py", line 62, in acquire
2018-05-30T15:42:20.213742+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) callback(*partial_args, **partial_kwargs)
2018-05-30T15:42:20.213789+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 226, in _process_task
2018-05-30T15:42:20.213849+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) req.execute_using_pool(self.pool)
2018-05-30T15:42:20.213896+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/worker/request.py", line 531, in execute_using_pool
2018-05-30T15:42:20.213942+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) correlation_id=task_id,
2018-05-30T15:42:20.214041+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib/python2.7/site-packages/celery/concurrency/base.py", line 155, in apply_async
2018-05-30T15:42:20.214088+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) **options)
2018-05-30T15:42:20.214133+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1486, in apply_async
2018-05-30T15:42:20.214178+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) self._quick_put((TASK, (result._job, None, func, args, kwds)))
2018-05-30T15:42:20.214226+00:00 pulp1.infra-ng.dev pulp[21584]: celery.worker:CRITICAL: (21584-98112) TypeError: 'NoneType' object is not callable
2018-05-30T15:42:21.734082+00:00 pulp1.infra-ng.dev pulp[21584]: pulp.server.async.tasks:INFO: Cleaning up shutdown worker 'resource_manager@pulp.ciscolabs.infra-ng.dev'.
2018-05-30T15:42:25.770581+00:00 pulp1.infra-ng.dev pulp[845]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
    (1-1/1)