Nov 15 19:44:17 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:17.531972 Nov 15 19:44:19 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.unit_association.unassociate_by_criteria[349653b2-6f30-46f3-9e34-0e06c03fa2e7] succeeded in 211.595463248s: {'units_successful': [{'unit_key': {'id': u'RHEA-2016:0026'}, 'type_id': 'erratum'}, {'unit_key': {'id': u'RHBA-2015:0329'},... Nov 15 19:44:20 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[2f189674-805d-4568-aa30-43b35caac650] succeeded in 0.838325507008s: None Nov 15 19:44:20 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:20.919109 Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998588 Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998567 Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998598 Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998616 Nov 15 19:44:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:22.034755 Nov 15 19:44:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:22.239038 Nov 15 19:44:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:22.512447 Nov 15 19:44:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:23.060028 Nov 15 19:44:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:23.062187 Nov 15 19:44:26 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:26.752387 Nov 15 19:44:27 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:27.190659 Nov 15 19:44:27 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:27.190659 Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.405788 Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.405823 Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.681794 Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.681792 Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.682214 Nov 15 19:44:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:34.137913 Nov 15 19:44:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:34.138076 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.092036 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.092061 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495396 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495449 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495465 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495589 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.593825 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.594376 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.594521 Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.595292 Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670056 Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670091 Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670193 Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670274 Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.671341 Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.671428 Nov 15 19:44:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:45.159562 Nov 15 19:44:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:45.306860 Nov 15 19:44:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:45.836708 Nov 15 19:44:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:47.097608 Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322017 Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322036 Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322432 Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322883 Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.323432 Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.669127 Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.696709 Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.696728 Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.697153 Nov 15 19:44:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:52.795993 Nov 15 19:44:55 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:55.745626 Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.961585 Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.961601 Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.962241 Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.962870 Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.076437 Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.465025 Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.465490 Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.465675 Nov 15 19:45:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:01.052704 Nov 15 19:45:01 sat-r220-02 systemd: Started Session 30919 of user root. Nov 15 19:45:01 sat-r220-02 systemd: Starting Session 30919 of user root. Nov 15 19:45:02 sat-r220-02 su: (to foreman) root on none Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.110631 Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.110685 Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.431449 Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.431439 Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.793204 Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.044605 Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.045137 Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.347531 Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.347838 Nov 15 19:45:11 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:11.978026 Nov 15 19:45:11 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:11.978104 Nov 15 19:45:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:12.679166 Nov 15 19:45:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:12.679174 Nov 15 19:45:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:12.681256 Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.720833 Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.721210 Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.869874 Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.915516 Nov 15 19:45:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:14.019360 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.227901 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.227828 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.228012 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.228195 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.228728 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.229397 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.394199 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.480591 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.624584 Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.625132 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.007616 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.007906 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.687214 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.862620 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.862620 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.862988 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.896994 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.897138 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.923422 Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.923439 Nov 15 19:45:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:30.963958 Nov 15 19:45:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:31.089236 Nov 15 19:45:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:31.089322 Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.284052 Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.394437 Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.395852 Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.396826 Nov 15 19:45:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:34.317729 Nov 15 19:45:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:34.317793 Nov 15 19:45:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:34.317817 Nov 15 19:45:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:36.384891 Nov 15 19:45:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:37.816056 Nov 15 19:45:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:37.816225 Nov 15 19:45:38 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:38.870055 Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.276137 Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.276253 Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.276291 Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.278326 Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.278501 Nov 15 19:45:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:41.614387 Nov 15 19:45:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:42.324055 Nov 15 19:45:43 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:43.988138 Nov 15 19:45:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:44.200767 Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.493323 Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.494635 Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.876076 Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.876216 Nov 15 19:45:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:46.418595 Nov 15 19:45:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:46.983045 Nov 15 19:45:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:48.638636 Nov 15 19:45:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:48.639524 Nov 15 19:45:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:49.722002 Nov 15 19:45:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:49.722914 Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.018015 Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.253881 Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.368484 Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.368490 Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.368579 Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.369886 Nov 15 19:45:54 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:54.909996 Nov 15 19:45:55 sat-r220-02 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Nov 15 19:45:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:56.019763 Nov 15 19:45:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:56.020455 Nov 15 19:45:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:56.021042 Nov 15 19:45:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:57.526040 Nov 15 19:45:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:57.526124 Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.057988 Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.058022 Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.058082 Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.059421 Nov 15 19:46:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:01.270547 Nov 15 19:46:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:02.013827 Nov 15 19:46:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:03.681747 Nov 15 19:46:04 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:04.382106 Nov 15 19:46:04 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:04.382095 Nov 15 19:46:05 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:05.177255 Nov 15 19:46:07 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:07.718830 Nov 15 19:46:07 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:07.718858 Nov 15 19:46:07 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:07.831429 Nov 15 19:46:09 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:09.065196 Nov 15 19:46:09 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:09.065177 Nov 15 19:46:09 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:09.631107 Nov 15 19:46:10 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:10.712123 Nov 15 19:46:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:12.870624 Nov 15 19:46:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:12.871737 Nov 15 19:46:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:13.174211 Nov 15 19:46:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:13.175360 Nov 15 19:46:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:14.215520 Nov 15 19:46:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:14.330411 Nov 15 19:46:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:14.760952 Nov 15 19:46:15 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:15.414643 Nov 15 19:46:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:16.951112 Nov 15 19:46:17 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:17.605402 Nov 15 19:46:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:18.276963 Nov 15 19:46:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:18.277729 Nov 15 19:46:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:18.736900 Nov 15 19:46:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:19.375742 Nov 15 19:46:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:21.461992 Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.605667 Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.605693 Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.725664 Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.725863 Nov 15 19:46:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:23.293036 Nov 15 19:46:24 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:24.278837 Nov 15 19:46:24 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:24.503528 Nov 15 19:46:24 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:24.633425 Nov 15 19:46:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:25.801593 Nov 15 19:46:28 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:28.688410 Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264440 Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264475 Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264756 Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264992 Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.265507 Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.823927 Nov 15 19:46:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:31.082858 Nov 15 19:46:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:31.083719 Nov 15 19:46:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:31.579967 Nov 15 19:46:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:34.800487 Nov 15 19:46:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:34.956908 Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.625504 Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.625766 Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.698476 Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.698488 Nov 15 19:46:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:36.607731 Nov 15 19:46:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:36.607840 Nov 15 19:46:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:37.369195 Nov 15 19:46:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:37.369219 Nov 15 19:46:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:40.942188 Nov 15 19:46:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:40.942447 Nov 15 19:46:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:40.943013 Nov 15 19:46:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:41.035557 Nov 15 19:46:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:41.945136 Nov 15 19:46:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:42.434158 Nov 15 19:46:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:42.686951 Nov 15 19:46:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:42.689613 Nov 15 19:46:43 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:43.480830 Nov 15 19:46:43 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:43.481256 Nov 15 19:46:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:46.505478 Nov 15 19:46:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:46.505524 Nov 15 19:46:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:47.333918 Nov 15 19:46:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:47.408413 Nov 15 19:46:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:47.960570 Nov 15 19:46:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:48.010024 Nov 15 19:46:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:48.178378 Nov 15 19:46:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:48.179688 Nov 15 19:46:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:49.538909 Nov 15 19:46:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:49.539219 Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.192578 Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.192674 Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.192918 Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.193884 Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.193977 Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.637870 Nov 15 19:46:54 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:54.660994 Nov 15 19:46:54 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:54.660995 Nov 15 19:46:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:56.151603 Nov 15 19:46:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:57.702088 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444842 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444678 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444754 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444753 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.869004 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.909035 Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.992403 Nov 15 19:47:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:21.080578 Nov 15 19:47:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:21.080599 Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.154546 Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.154547 Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.279477 Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.279534 Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:46:53.192000+00:00 / 2017-11-16 00:46:56.588725+00:00 Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.281951 Nov 15 19:47:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:44.170680 Nov 15 19:47:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:44.170680 Nov 15 19:47:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:45.573458 Nov 15 19:47:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:45.720894 Nov 15 19:47:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:47.634029 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.448013 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.448002 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.500323 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.500571 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.500809 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.501786 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.502725 Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.749910 Nov 15 19:47:52 sat-r220-02 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Nov 15 19:47:52 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 387cd591-eca5-49e0-ae6d-e074ecc88362 (15) Nov 15 19:47:58 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 387cd591-eca5-49e0-ae6d-e074ecc88362. Nov 15 19:48:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:06.490427 Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.512820 Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.513010 Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.649179 Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.649179 Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.649178 Nov 15 19:48:20 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c618e7f2-bfcf-4c35-8175-a8b8f2d981ba] Nov 15 19:48:21 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 03ee5e46-6bab-4bb2-9f49-93a3f48844ee. Nov 15 19:48:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:21.167777 Nov 15 19:48:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:21.648014 Nov 15 19:48:21 sat-r220-02 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Nov 15 19:48:28 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 03ee5e46-6bab-4bb2-9f49-93a3f48844ee (15) Nov 15 19:48:28 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 840c5829-1c3c-44f8-9e63-630fbf0d8994 (15) Nov 15 19:48:37 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 840c5829-1c3c-44f8-9e63-630fbf0d8994. Nov 15 19:48:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:39.652018 Nov 15 19:48:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:40.535881 Nov 15 19:48:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:47.706969 Nov 15 19:48:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:47.707064 Nov 15 19:48:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:47.708065 Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 8aaf5a29-b8de-41c3-a0f4-88f3a9491ec3. Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:51.164485 Nov 15 19:48:51 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 8aaf5a29-b8de-41c3-a0f4-88f3a9491ec3 (15) Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:51.823060 Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:51.823060 Nov 15 19:48:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:57.938882 Nov 15 19:49:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:02.431348 Nov 15 19:49:02 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 55d0ce80-03c7-4871-99c2-31f523c2a672. Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: a72f86b4-b908-49e7-9f73-4c46c15a297e. Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.368487 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.368627 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.368840 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.370240 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.787405 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.787408 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.787408 Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.788405 Nov 15 19:49:19 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[8b7f2786-218a-4c08-93f5-30e7d03fea82] Nov 15 19:49:19 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[05914dab-dc83-4c08-9875-5574277d232d] Nov 15 19:49:21 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: e8d231b2-0a05-4924-bc2d-ba943e08ce10. Nov 15 19:49:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:21.901819 Nov 15 19:49:21 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 55d0ce80-03c7-4871-99c2-31f523c2a672 (15) Nov 15 19:49:21 sat-r220-02 pulp: celery.worker.control:INFO: Terminating a72f86b4-b908-49e7-9f73-4c46c15a297e (15) Nov 15 19:49:21 sat-r220-02 pulp: celery.worker.control:INFO: Terminating e8d231b2-0a05-4924-bc2d-ba943e08ce10 (15) Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) Task pulp.server.managers.repo.unit_association.unassociate_by_criteria[e8d231b2-0a05-4924-bc2d-ba943e08ce10] raised unexpected: Terminated(15,) Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) Traceback (most recent call last): Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) raise Terminated(-(signum or 0)) Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) Terminated: 15 Nov 15 19:49:23 sat-r220-02 celery: reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com ready. Nov 15 19:49:23 sat-r220-02 pulp: py.warnings:WARNING: [b836e646] (12627-26336) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing> Nov 15 19:49:23 sat-r220-02 pulp: py.warnings:WARNING: [b836e646] (12627-26336) "MongoClient opened before fork. Create MongoClient " Nov 15 19:49:23 sat-r220-02 pulp: py.warnings:WARNING: [b836e646] (12627-26336) Nov 15 19:49:27 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c618e7f2-bfcf-4c35-8175-a8b8f2d981ba] succeeded in 67.008401799s: None Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.106383 Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.106467 Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.139658 Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.141934 Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.143196 Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.497409 Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.498019 Nov 15 19:49:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:39.791874 Nov 15 19:49:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:41.266909 Nov 15 19:49:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:51.681696 Nov 15 19:49:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:51.683005 Nov 15 19:49:51 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[840c5829-1c3c-44f8-9e63-630fbf0d8994] Nov 15 19:49:51 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b836e646-a7dc-4c7b-af29-58056741045f] succeeded in 28.8063739622s: None Nov 15 19:49:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:51.745816 Nov 15 19:49:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:52.349315 Nov 15 19:49:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:52.630125 Nov 15 19:49:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:53.277169 Nov 15 19:49:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:53.486621 Nov 15 19:49:54 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[03ee5e46-6bab-4bb2-9f49-93a3f48844ee] Nov 15 19:49:54 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9f91f9df-f46e-4d3d-a38e-7f1c566353a1] succeeded in 2.6269860575s: None Nov 15 19:49:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:56.667279 Nov 15 19:49:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:57.076997 Nov 15 19:49:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:57.077131 Nov 15 19:49:57 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[55d0ce80-03c7-4871-99c2-31f523c2a672] Nov 15 19:49:57 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[32c29d33-0d57-4c88-a516-d53d4b0cb50b] succeeded in 3.50428021699s: None Nov 15 19:49:58 sat-r220-02 pulp: py.warnings:WARNING: (28726-20096) /usr/lib/python2.7/site-packages/pulp/server/managers/schedule/utils.py:118: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead. Nov 15 19:49:58 sat-r220-02 pulp: py.warnings:WARNING: (28726-20096) ScheduledCall.get_collection().remove({'resource': resource}) Nov 15 19:49:58 sat-r220-02 pulp: py.warnings:WARNING: (28726-20096) Nov 15 19:50:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:01.178099 Nov 15 19:50:01 sat-r220-02 systemd: Started Session 30920 of user root. Nov 15 19:50:01 sat-r220-02 systemd: Starting Session 30920 of user root. Nov 15 19:50:02 sat-r220-02 su: (to foreman) root on none Nov 15 19:50:11 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:11.305508 Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.014085 Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.106642 Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.780477 Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.991760 Nov 15 19:50:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:23.898696 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.227683 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.322340 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.345501 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.346920 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.346963 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:49:57.077000+00:00 / 2017-11-16 00:50:02.112930+00:00 Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:50:51 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[8aaf5a29-b8de-41c3-a0f4-88f3a9491ec3] Nov 15 19:50:51 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[dbf30d37-3a91-47ce-98e3-9a30210bd85b] succeeded in 53.7210023366s: None Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.131009 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.542199 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.542135 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.542595 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.544750 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.545188 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.545716 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.545822 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.546082 Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.548828 Nov 15 19:50:56 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[387cd591-eca5-49e0-ae6d-e074ecc88362] Nov 15 19:50:56 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[289f57bf-2837-4b4e-a5bd-2bf1501b6505] succeeded in 5.31645996869s: None Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 148504e5-1379-4140-869a-5db6b31ddbff (15) Nov 15 19:50:57 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 148504e5-1379-4140-869a-5db6b31ddbff. Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[bbf23ea8-8f27-44b1-910c-77d5d4c7a4d6] Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Task pulp.server.managers.repo.unit_association.associate_from_repo[148504e5-1379-4140-869a-5db6b31ddbff] raised unexpected: Terminated(15,) Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Traceback (most recent call last): Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) raise Terminated(-(signum or 0)) Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Terminated: 15 Nov 15 19:50:57 sat-r220-02 celery: reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com ready. Nov 15 19:50:57 sat-r220-02 pulp: py.warnings:WARNING: [f8cd5051] (12800-85088) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing> Nov 15 19:50:57 sat-r220-02 pulp: py.warnings:WARNING: [f8cd5051] (12800-85088) "MongoClient opened before fork. Create MongoClient " Nov 15 19:50:57 sat-r220-02 pulp: py.warnings:WARNING: [f8cd5051] (12800-85088) Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[a72f86b4-b908-49e7-9f73-4c46c15a297e] Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[6052d289-5217-495a-8d17-69d83ccec440] succeeded in 0.592279043049s: None Nov 15 19:50:59 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[f8cd5051-4ba9-4e1f-941c-80cbe60d70d7] succeeded in 2.16901315842s: None Nov 15 19:50:59 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[1c669292-ef33-4a8c-9a90-36ed92f1a4ee] succeeded in 2.31580980774s: None Nov 15 19:51:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:02.233405 Nov 15 19:51:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:02.347814 Nov 15 19:51:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:02.347824 Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907466 Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907485 Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907772 Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907841 Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.909216 Nov 15 19:51:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:14.765665 Nov 15 19:51:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:14.767170 Nov 15 19:51:17 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[bbf23ea8-8f27-44b1-910c-77d5d4c7a4d6] succeeded in 19.9465367226s: None Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.797786 Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.797938 Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.798215 Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.799555 Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.799906 Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.800206 Nov 15 19:51:40 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[f01a25e8-5fb1-419d-9b01-17d494d6d9f8] Nov 15 19:51:40 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[541dab26-2ffc-4b66-8d2d-1c494559d1ca] Nov 15 19:51:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:45.044156 Nov 15 19:51:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:47.510670 Nov 15 19:51:47 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:51:13.907000+00:00 / 2017-11-16 00:51:19.801492+00:00 Nov 15 19:51:47 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.030702 Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.030793 Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.030925 Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.334758 Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.284062 Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.284146 Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.395606 Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.396946 Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 8b7f2786-218a-4c08-93f5-30e7d03fea82 (15) Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) Task pulp.server.tasks.repository.delete[8b7f2786-218a-4c08-93f5-30e7d03fea82] raised unexpected: Terminated(15,) Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) Traceback (most recent call last): Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated Nov 15 19:51:49 sat-r220-02 celery: reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com ready. Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) raise Terminated(-(signum or 0)) Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) Terminated: 15 Nov 15 19:51:49 sat-r220-02 pulp: py.warnings:WARNING: [05914dab] (12869-20096) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing> Nov 15 19:51:49 sat-r220-02 pulp: py.warnings:WARNING: [05914dab] (12869-20096) "MongoClient opened before fork. Create MongoClient " Nov 15 19:51:49 sat-r220-02 pulp: py.warnings:WARNING: [05914dab] (12869-20096) Nov 15 19:51:49 sat-r220-02 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred) Nov 15 19:51:51 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 8b7f2786-218a-4c08-93f5-30e7d03fea82. Nov 15 19:51:51 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:51:14.767000+00:00 / 2017-11-16 00:51:19.801492+00:00 Nov 15 19:51:51 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.596815 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.753106 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.753358 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.753695 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.837077 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.837110 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.885470 Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.919830 Nov 15 19:52:01 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[96842eec-196d-4367-8f35-fd2d3c0d3e36] Nov 15 19:52:01 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[05914dab-dc83-4c08-9875-5574277d232d] succeeded in 12.4820518587s: None Nov 15 19:52:02 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 7de447e9-0f08-4dd7-9c9d-ca5409f0f46b (15) Nov 15 19:52:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:22.106769 Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.345548 Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.345534 Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.345557 Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.346006 Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.346028 Nov 15 19:52:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:45.787075 Nov 15 19:52:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:45.787217 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132246 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132708 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132727 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132862 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132895 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.133633 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 7de447e9-0f08-4dd7-9c9d-ca5409f0f46b. Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.403596 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:51:14.765000+00:00 / 2017-11-16 00:51:19.801492+00:00 Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:52:46 sat-r220-02 pulp: py.warnings:WARNING: (12800-85088) /usr/lib/python2.7/site-packages/pulp/server/managers/schedule/utils.py:118: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead. Nov 15 19:52:46 sat-r220-02 pulp: py.warnings:WARNING: (12800-85088) ScheduledCall.get_collection().remove({'resource': resource}) Nov 15 19:52:46 sat-r220-02 pulp: py.warnings:WARNING: (12800-85088) Nov 15 19:52:49 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c50a2d83-289f-4bee-941e-d56a77717672] Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462142 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462155 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462208 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462178 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462861 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462980 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.463809 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.463898 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.463873 Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.465081 Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.control:INFO: Terminating f01a25e8-5fb1-419d-9b01-17d494d6d9f8 (15) Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Task pulp.server.tasks.repository.delete[f01a25e8-5fb1-419d-9b01-17d494d6d9f8] raised unexpected: Terminated(15,) Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Traceback (most recent call last): Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) raise Terminated(-(signum or 0)) Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Terminated: 15 Nov 15 19:53:12 sat-r220-02 pulp: py.warnings:WARNING: [541dab26] (12968-85088) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing> Nov 15 19:53:12 sat-r220-02 pulp: py.warnings:WARNING: [541dab26] (12968-85088) "MongoClient opened before fork. Create MongoClient " Nov 15 19:53:12 sat-r220-02 pulp: py.warnings:WARNING: [541dab26] (12968-85088) Nov 15 19:53:20 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:20.826412 Nov 15 19:53:20 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:20.827871 Nov 15 19:53:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:25.978110 Nov 15 19:53:26 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: f01a25e8-5fb1-419d-9b01-17d494d6d9f8. Nov 15 19:53:26 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:26.397730 Nov 15 19:53:26 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c50a2d83-289f-4bee-941e-d56a77717672] succeeded in 37.077276946s: None Nov 15 19:53:27 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:27.992563 Nov 15 19:53:32 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:32.846345 Nov 15 19:53:35 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker scheduler@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.462000+00:00 / 2017-11-16 00:53:06.339549+00:00 Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.465000+00:00 / 2017-11-16 00:53:06.339549+00:00 Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.287075 Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.287722 Nov 15 19:53:36 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[541dab26-2ffc-4b66-8d2d-1c494559d1ca] succeeded in 23.6107569775s: None Nov 15 19:53:36 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[ace87344-b537-4f41-be02-2df46bc26af5] Nov 15 19:53:36 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[39445f3f-39cd-47fb-a9c2-84c9ca713a9a] Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.425234 Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.479119 Nov 15 19:53:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:47.349187 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.315762 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.368979 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.369100 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.769618 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.770920 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.463000+00:00 / 2017-11-16 00:53:06.339549+00:00 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.771069 Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:53:54 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 53f6365c-7aaf-4cdb-a28b-af988c6d22b9 (15) Nov 15 19:53:58 sat-r220-02 puppet-master[29050]: Compiled catalog for sat-r220-02.lab.eng.rdu2.redhat.com in environment production in 0.01 seconds Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 53f6365c-7aaf-4cdb-a28b-af988c6d22b9. Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.745846 Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.745948 Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' discovered Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.747478 Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.747467 Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.748681 Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: ced18cf7-e816-4abf-a365-bacb1cdda736. Nov 15 19:53:58 sat-r220-02 pulp: celery.worker.control:INFO: Terminating ced18cf7-e816-4abf-a365-bacb1cdda736 (15) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.029131 Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 30073c56-cf84-482d-b279-7eb0ebe82ba6 (15) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 30073c56-cf84-482d-b279-7eb0ebe82ba6. Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: (12968-85088) /usr/lib/python2.7/site-packages/pulp/server/managers/schedule/utils.py:118: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead. Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: (12968-85088) ScheduledCall.get_collection().remove({'resource': resource}) Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: (12968-85088) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.360683 Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.361567 Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 4cc2dc28-af1b-4922-9307-881368dbb3dd (15) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 4cc2dc28-af1b-4922-9307-881368dbb3dd. Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating d188c7cc-5050-466c-93b8-b0d207472567 (15) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.754305 Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: d188c7cc-5050-466c-93b8-b0d207472567. Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 91b0c678-85d5-4b07-93ac-8276496016aa (15) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 91b0c678-85d5-4b07-93ac-8276496016aa. Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 0cde6a92-765e-4219-9365-c4cc18436b0a (15) Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) Task pulp.server.managers.repo.unit_association.unassociate_by_criteria[0cde6a92-765e-4219-9365-c4cc18436b0a] raised unexpected: Terminated(15,) Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) Traceback (most recent call last): Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) raise Terminated(-(signum or 0)) Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) Terminated: 15 Nov 15 19:53:59 sat-r220-02 celery: reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com ready. Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: [8500f39b] (13272-43872) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing> Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: [8500f39b] (13272-43872) "MongoClient opened before fork. Create MongoClient " Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: [8500f39b] (13272-43872) Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 0cde6a92-765e-4219-9365-c4cc18436b0a. Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.462000+00:00 / 2017-11-16 00:53:06.339549+00:00 Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue. Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[4cc2dc28-af1b-4922-9307-881368dbb3dd] Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8500f39b-f9b1-4039-a2b3-6e745fc9dbcb] succeeded in 0.12942882441s: None Nov 15 19:54:00 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running. Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[30073c56-cf84-482d-b279-7eb0ebe82ba6] Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[2b78a57a-a87a-4d83-b44c-f70c1158e8ce] succeeded in 0.0994412815198s: None Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[53f6365c-7aaf-4cdb-a28b-af988c6d22b9] Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b2660f0c-43e1-4d30-a7ca-ae3575db68af] succeeded in 0.275608449243s: None Nov 15 19:54:00 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:542: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead. Nov 15 19:54:00 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) RepoSyncResult.get_collection().remove({'repo_id': repo_id}) Nov 15 19:54:00 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:543: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead. Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) RepoPublishResult.get_collection().remove({'repo_id': repo_id}) Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [36c8cd54] (13272-43872) /usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead. Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [36c8cd54] (13272-43872) upsert=upsert, **write_concern) Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [36c8cd54] (13272-43872)