Project

Profile

Help

Story #3135 » 532826.txt

More logging with heartbeats recorded - daviddavis, 11/16/2017 03:34 PM

 
Nov 15 19:44:17 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:17.531972
Nov 15 19:44:19 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.unit_association.unassociate_by_criteria[349653b2-6f30-46f3-9e34-0e06c03fa2e7] succeeded in 211.595463248s: {'units_successful': [{'unit_key': {'id': u'RHEA-2016:0026'}, 'type_id': 'erratum'}, {'unit_key': {'id': u'RHBA-2015:0329'},...
Nov 15 19:44:20 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[2f189674-805d-4568-aa30-43b35caac650] succeeded in 0.838325507008s: None
Nov 15 19:44:20 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:20.919109
Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998588
Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998567
Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998598
Nov 15 19:44:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:21.998616
Nov 15 19:44:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:22.034755
Nov 15 19:44:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:22.239038
Nov 15 19:44:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:22.512447
Nov 15 19:44:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:23.060028
Nov 15 19:44:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:23.062187
Nov 15 19:44:26 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:26.752387
Nov 15 19:44:27 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:27.190659
Nov 15 19:44:27 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:27.190659
Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.405788
Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.405823
Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.681794
Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.681792
Nov 15 19:44:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:33.682214
Nov 15 19:44:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:34.137913
Nov 15 19:44:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:34.138076
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.092036
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.092061
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495396
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495449
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495465
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.495589
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.593825
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.594376
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.594521
Nov 15 19:44:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:39.595292
Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670056
Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670091
Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670193
Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.670274
Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.671341
Nov 15 19:44:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:44.671428
Nov 15 19:44:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:45.159562
Nov 15 19:44:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:45.306860
Nov 15 19:44:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:45.836708
Nov 15 19:44:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:47.097608
Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322017
Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322036
Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322432
Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.322883
Nov 15 19:44:50 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:50.323432
Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.669127
Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.696709
Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.696728
Nov 15 19:44:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:51.697153
Nov 15 19:44:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:52.795993
Nov 15 19:44:55 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:55.745626
Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.961585
Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.961601
Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.962241
Nov 15 19:44:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:56.962870
Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.076437
Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.465025
Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.465490
Nov 15 19:44:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:44:59.465675
Nov 15 19:45:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:01.052704
Nov 15 19:45:01 sat-r220-02 systemd: Started Session 30919 of user root.
Nov 15 19:45:01 sat-r220-02 systemd: Starting Session 30919 of user root.
Nov 15 19:45:02 sat-r220-02 su: (to foreman) root on none
Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.110631
Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.110685
Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.431449
Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.431439
Nov 15 19:45:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:03.793204
Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.044605
Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.045137
Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.347531
Nov 15 19:45:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:06.347838
Nov 15 19:45:11 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:11.978026
Nov 15 19:45:11 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:11.978104
Nov 15 19:45:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:12.679166
Nov 15 19:45:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:12.679174
Nov 15 19:45:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:12.681256
Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.720833
Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.721210
Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.869874
Nov 15 19:45:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:13.915516
Nov 15 19:45:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:14.019360
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.227901
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.227828
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.228012
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.228195
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.228728
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.229397
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.394199
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.480591
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.624584
Nov 15 19:45:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:19.625132
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.007616
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.007906
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.687214
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.862620
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.862620
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.862988
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.896994
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.897138
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.923422
Nov 15 19:45:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:25.923439
Nov 15 19:45:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:30.963958
Nov 15 19:45:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:31.089236
Nov 15 19:45:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:31.089322
Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.284052
Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.394437
Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.395852
Nov 15 19:45:33 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:33.396826
Nov 15 19:45:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:34.317729
Nov 15 19:45:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:34.317793
Nov 15 19:45:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:34.317817
Nov 15 19:45:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:36.384891
Nov 15 19:45:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:37.816056
Nov 15 19:45:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:37.816225
Nov 15 19:45:38 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:38.870055
Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.276137
Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.276253
Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.276291
Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.278326
Nov 15 19:45:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:40.278501
Nov 15 19:45:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:41.614387
Nov 15 19:45:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:42.324055
Nov 15 19:45:43 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:43.988138
Nov 15 19:45:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:44.200767
Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.493323
Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.494635
Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.876076
Nov 15 19:45:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:45.876216
Nov 15 19:45:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:46.418595
Nov 15 19:45:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:46.983045
Nov 15 19:45:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:48.638636
Nov 15 19:45:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:48.639524
Nov 15 19:45:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:49.722002
Nov 15 19:45:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:49.722914
Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.018015
Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.253881
Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.368484
Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.368490
Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.368579
Nov 15 19:45:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:52.369886
Nov 15 19:45:54 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:54.909996
Nov 15 19:45:55 sat-r220-02 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Nov 15 19:45:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:56.019763
Nov 15 19:45:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:56.020455
Nov 15 19:45:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:56.021042
Nov 15 19:45:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:57.526040
Nov 15 19:45:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:57.526124
Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.057988
Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.058022
Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.058082
Nov 15 19:45:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:45:59.059421
Nov 15 19:46:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:01.270547
Nov 15 19:46:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:02.013827
Nov 15 19:46:03 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:03.681747
Nov 15 19:46:04 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:04.382106
Nov 15 19:46:04 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:04.382095
Nov 15 19:46:05 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:05.177255
Nov 15 19:46:07 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:07.718830
Nov 15 19:46:07 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:07.718858
Nov 15 19:46:07 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:07.831429
Nov 15 19:46:09 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:09.065196
Nov 15 19:46:09 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:09.065177
Nov 15 19:46:09 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:09.631107
Nov 15 19:46:10 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:10.712123
Nov 15 19:46:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:12.870624
Nov 15 19:46:12 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:12.871737
Nov 15 19:46:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:13.174211
Nov 15 19:46:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:13.175360
Nov 15 19:46:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:14.215520
Nov 15 19:46:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:14.330411
Nov 15 19:46:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:14.760952
Nov 15 19:46:15 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:15.414643
Nov 15 19:46:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:16.951112
Nov 15 19:46:17 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:17.605402
Nov 15 19:46:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:18.276963
Nov 15 19:46:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:18.277729
Nov 15 19:46:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:18.736900
Nov 15 19:46:19 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:19.375742
Nov 15 19:46:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:21.461992
Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.605667
Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.605693
Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.725664
Nov 15 19:46:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:22.725863
Nov 15 19:46:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:23.293036
Nov 15 19:46:24 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:24.278837
Nov 15 19:46:24 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:24.503528
Nov 15 19:46:24 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:24.633425
Nov 15 19:46:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:25.801593
Nov 15 19:46:28 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:28.688410
Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264440
Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264475
Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264756
Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.264992
Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.265507
Nov 15 19:46:29 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:29.823927
Nov 15 19:46:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:31.082858
Nov 15 19:46:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:31.083719
Nov 15 19:46:31 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:31.579967
Nov 15 19:46:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:34.800487
Nov 15 19:46:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:34.956908
Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.625504
Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.625766
Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.698476
Nov 15 19:46:35 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:35.698488
Nov 15 19:46:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:36.607731
Nov 15 19:46:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:36.607840
Nov 15 19:46:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:37.369195
Nov 15 19:46:37 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:37.369219
Nov 15 19:46:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:40.942188
Nov 15 19:46:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:40.942447
Nov 15 19:46:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:40.943013
Nov 15 19:46:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:41.035557
Nov 15 19:46:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:41.945136
Nov 15 19:46:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:42.434158
Nov 15 19:46:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:42.686951
Nov 15 19:46:42 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:42.689613
Nov 15 19:46:43 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:43.480830
Nov 15 19:46:43 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:43.481256
Nov 15 19:46:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:46.505478
Nov 15 19:46:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:46.505524
Nov 15 19:46:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:47.333918
Nov 15 19:46:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:47.408413
Nov 15 19:46:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:47.960570
Nov 15 19:46:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:48.010024
Nov 15 19:46:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:48.178378
Nov 15 19:46:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:48.179688
Nov 15 19:46:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:49.538909
Nov 15 19:46:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:49.539219
Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.192578
Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.192674
Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.192918
Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.193884
Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.193977
Nov 15 19:46:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:53.637870
Nov 15 19:46:54 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:54.660994
Nov 15 19:46:54 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:54.660995
Nov 15 19:46:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:56.151603
Nov 15 19:46:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:46:57.702088
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444842
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444678
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444754
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.444753
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.869004
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.909035
Nov 15 19:47:16 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:16.992403
Nov 15 19:47:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:21.080578
Nov 15 19:47:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:21.080599
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.154546
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.154547
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.279477
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.279534
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:46:53.192000+00:00 / 2017-11-16 00:46:56.588725+00:00
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:47:34 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:34.281951
Nov 15 19:47:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:44.170680
Nov 15 19:47:44 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:44.170680
Nov 15 19:47:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:45.573458
Nov 15 19:47:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:45.720894
Nov 15 19:47:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:47.634029
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.448013
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.448002
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.500323
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.500571
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.500809
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.501786
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.502725
Nov 15 19:47:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:47:51.749910
Nov 15 19:47:52 sat-r220-02 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Nov 15 19:47:52 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 387cd591-eca5-49e0-ae6d-e074ecc88362 (15)
Nov 15 19:47:58 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 387cd591-eca5-49e0-ae6d-e074ecc88362.
Nov 15 19:48:06 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:06.490427
Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.512820
Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.513010
Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.649179
Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.649179
Nov 15 19:48:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:18.649178
Nov 15 19:48:20 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c618e7f2-bfcf-4c35-8175-a8b8f2d981ba]
Nov 15 19:48:21 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 03ee5e46-6bab-4bb2-9f49-93a3f48844ee.
Nov 15 19:48:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:21.167777
Nov 15 19:48:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:21.648014
Nov 15 19:48:21 sat-r220-02 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Nov 15 19:48:28 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 03ee5e46-6bab-4bb2-9f49-93a3f48844ee (15)
Nov 15 19:48:28 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 840c5829-1c3c-44f8-9e63-630fbf0d8994 (15)
Nov 15 19:48:37 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 840c5829-1c3c-44f8-9e63-630fbf0d8994.
Nov 15 19:48:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:39.652018
Nov 15 19:48:40 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:40.535881
Nov 15 19:48:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:47.706969
Nov 15 19:48:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:47.707064
Nov 15 19:48:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:47.708065
Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 8aaf5a29-b8de-41c3-a0f4-88f3a9491ec3.
Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:51.164485
Nov 15 19:48:51 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 8aaf5a29-b8de-41c3-a0f4-88f3a9491ec3 (15)
Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:51.823060
Nov 15 19:48:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:51.823060
Nov 15 19:48:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:48:57.938882
Nov 15 19:49:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:02.431348
Nov 15 19:49:02 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 55d0ce80-03c7-4871-99c2-31f523c2a672.
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: a72f86b4-b908-49e7-9f73-4c46c15a297e.
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.368487
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.368627
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.368840
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.370240
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.787405
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.787408
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.787408
Nov 15 19:49:18 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:18.788405
Nov 15 19:49:19 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[8b7f2786-218a-4c08-93f5-30e7d03fea82]
Nov 15 19:49:19 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[05914dab-dc83-4c08-9875-5574277d232d]
Nov 15 19:49:21 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: e8d231b2-0a05-4924-bc2d-ba943e08ce10.
Nov 15 19:49:21 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:21.901819
Nov 15 19:49:21 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 55d0ce80-03c7-4871-99c2-31f523c2a672 (15)
Nov 15 19:49:21 sat-r220-02 pulp: celery.worker.control:INFO: Terminating a72f86b4-b908-49e7-9f73-4c46c15a297e (15)
Nov 15 19:49:21 sat-r220-02 pulp: celery.worker.control:INFO: Terminating e8d231b2-0a05-4924-bc2d-ba943e08ce10 (15)
Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) Task pulp.server.managers.repo.unit_association.unassociate_by_criteria[e8d231b2-0a05-4924-bc2d-ba943e08ce10] raised unexpected: Terminated(15,)
Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) Traceback (most recent call last):
Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated
Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) raise Terminated(-(signum or 0))
Nov 15 19:49:22 sat-r220-02 pulp: celery.worker.job:ERROR: (28120-26336) Terminated: 15
Nov 15 19:49:23 sat-r220-02 celery: reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com ready.
Nov 15 19:49:23 sat-r220-02 pulp: py.warnings:WARNING: [b836e646] (12627-26336) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Nov 15 19:49:23 sat-r220-02 pulp: py.warnings:WARNING: [b836e646] (12627-26336) "MongoClient opened before fork. Create MongoClient "
Nov 15 19:49:23 sat-r220-02 pulp: py.warnings:WARNING: [b836e646] (12627-26336)
Nov 15 19:49:27 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c618e7f2-bfcf-4c35-8175-a8b8f2d981ba] succeeded in 67.008401799s: None
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.106383
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.106467
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.139658
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.141934
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.143196
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.497409
Nov 15 19:49:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:36.498019
Nov 15 19:49:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:39.791874
Nov 15 19:49:41 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:41.266909
Nov 15 19:49:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:51.681696
Nov 15 19:49:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:51.683005
Nov 15 19:49:51 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[840c5829-1c3c-44f8-9e63-630fbf0d8994]
Nov 15 19:49:51 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b836e646-a7dc-4c7b-af29-58056741045f] succeeded in 28.8063739622s: None
Nov 15 19:49:51 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:51.745816
Nov 15 19:49:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:52.349315
Nov 15 19:49:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:52.630125
Nov 15 19:49:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:53.277169
Nov 15 19:49:53 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:53.486621
Nov 15 19:49:54 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[03ee5e46-6bab-4bb2-9f49-93a3f48844ee]
Nov 15 19:49:54 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9f91f9df-f46e-4d3d-a38e-7f1c566353a1] succeeded in 2.6269860575s: None
Nov 15 19:49:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:56.667279
Nov 15 19:49:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:57.076997
Nov 15 19:49:57 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:49:57.077131
Nov 15 19:49:57 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[55d0ce80-03c7-4871-99c2-31f523c2a672]
Nov 15 19:49:57 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[32c29d33-0d57-4c88-a516-d53d4b0cb50b] succeeded in 3.50428021699s: None
Nov 15 19:49:58 sat-r220-02 pulp: py.warnings:WARNING: (28726-20096) /usr/lib/python2.7/site-packages/pulp/server/managers/schedule/utils.py:118: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Nov 15 19:49:58 sat-r220-02 pulp: py.warnings:WARNING: (28726-20096) ScheduledCall.get_collection().remove({'resource': resource})
Nov 15 19:49:58 sat-r220-02 pulp: py.warnings:WARNING: (28726-20096)
Nov 15 19:50:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:01.178099
Nov 15 19:50:01 sat-r220-02 systemd: Started Session 30920 of user root.
Nov 15 19:50:01 sat-r220-02 systemd: Starting Session 30920 of user root.
Nov 15 19:50:02 sat-r220-02 su: (to foreman) root on none
Nov 15 19:50:11 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:11.305508
Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.014085
Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.106642
Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.780477
Nov 15 19:50:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:22.991760
Nov 15 19:50:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:23.898696
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.227683
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.322340
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.345501
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.346920
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:30.346963
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:49:57.077000+00:00 / 2017-11-16 00:50:02.112930+00:00
Nov 15 19:50:30 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:50:51 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[8aaf5a29-b8de-41c3-a0f4-88f3a9491ec3]
Nov 15 19:50:51 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[dbf30d37-3a91-47ce-98e3-9a30210bd85b] succeeded in 53.7210023366s: None
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.131009
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.542199
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.542135
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.542595
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.544750
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.545188
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.545716
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.545822
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.546082
Nov 15 19:50:56 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:50:56.548828
Nov 15 19:50:56 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[387cd591-eca5-49e0-ae6d-e074ecc88362]
Nov 15 19:50:56 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[289f57bf-2837-4b4e-a5bd-2bf1501b6505] succeeded in 5.31645996869s: None
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 148504e5-1379-4140-869a-5db6b31ddbff (15)
Nov 15 19:50:57 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 148504e5-1379-4140-869a-5db6b31ddbff.
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[bbf23ea8-8f27-44b1-910c-77d5d4c7a4d6]
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Task pulp.server.managers.repo.unit_association.associate_from_repo[148504e5-1379-4140-869a-5db6b31ddbff] raised unexpected: Terminated(15,)
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Traceback (most recent call last):
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) raise Terminated(-(signum or 0))
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Terminated: 15
Nov 15 19:50:57 sat-r220-02 celery: reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com ready.
Nov 15 19:50:57 sat-r220-02 pulp: py.warnings:WARNING: [f8cd5051] (12800-85088) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Nov 15 19:50:57 sat-r220-02 pulp: py.warnings:WARNING: [f8cd5051] (12800-85088) "MongoClient opened before fork. Create MongoClient "
Nov 15 19:50:57 sat-r220-02 pulp: py.warnings:WARNING: [f8cd5051] (12800-85088)
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[a72f86b4-b908-49e7-9f73-4c46c15a297e]
Nov 15 19:50:57 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[6052d289-5217-495a-8d17-69d83ccec440] succeeded in 0.592279043049s: None
Nov 15 19:50:59 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[f8cd5051-4ba9-4e1f-941c-80cbe60d70d7] succeeded in 2.16901315842s: None
Nov 15 19:50:59 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[1c669292-ef33-4a8c-9a90-36ed92f1a4ee] succeeded in 2.31580980774s: None
Nov 15 19:51:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:02.233405
Nov 15 19:51:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:02.347814
Nov 15 19:51:02 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:02.347824
Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907466
Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907485
Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907772
Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.907841
Nov 15 19:51:13 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:13.909216
Nov 15 19:51:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:14.765665
Nov 15 19:51:14 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:14.767170
Nov 15 19:51:17 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[bbf23ea8-8f27-44b1-910c-77d5d4c7a4d6] succeeded in 19.9465367226s: None
Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.797786
Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.797938
Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.798215
Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.799555
Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.799906
Nov 15 19:51:39 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:39.800206
Nov 15 19:51:40 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[f01a25e8-5fb1-419d-9b01-17d494d6d9f8]
Nov 15 19:51:40 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[541dab26-2ffc-4b66-8d2d-1c494559d1ca]
Nov 15 19:51:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:45.044156
Nov 15 19:51:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:47.510670
Nov 15 19:51:47 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:51:13.907000+00:00 / 2017-11-16 00:51:19.801492+00:00
Nov 15 19:51:47 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.030702
Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.030793
Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.030925
Nov 15 19:51:48 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:48.334758
Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.284062
Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.284146
Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.395606
Nov 15 19:51:49 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:51:49.396946
Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 8b7f2786-218a-4c08-93f5-30e7d03fea82 (15)
Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) Task pulp.server.tasks.repository.delete[8b7f2786-218a-4c08-93f5-30e7d03fea82] raised unexpected: Terminated(15,)
Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) Traceback (most recent call last):
Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated
Nov 15 19:51:49 sat-r220-02 celery: reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com ready.
Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) raise Terminated(-(signum or 0))
Nov 15 19:51:49 sat-r220-02 pulp: celery.worker.job:ERROR: (28114-20096) Terminated: 15
Nov 15 19:51:49 sat-r220-02 pulp: py.warnings:WARNING: [05914dab] (12869-20096) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Nov 15 19:51:49 sat-r220-02 pulp: py.warnings:WARNING: [05914dab] (12869-20096) "MongoClient opened before fork. Create MongoClient "
Nov 15 19:51:49 sat-r220-02 pulp: py.warnings:WARNING: [05914dab] (12869-20096)
Nov 15 19:51:49 sat-r220-02 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Nov 15 19:51:51 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 8b7f2786-218a-4c08-93f5-30e7d03fea82.
Nov 15 19:51:51 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:51:14.767000+00:00 / 2017-11-16 00:51:19.801492+00:00
Nov 15 19:51:51 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.596815
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.753106
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.753358
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.753695
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.837077
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.837110
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.885470
Nov 15 19:52:01 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:01.919830
Nov 15 19:52:01 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[96842eec-196d-4367-8f35-fd2d3c0d3e36]
Nov 15 19:52:01 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[05914dab-dc83-4c08-9875-5574277d232d] succeeded in 12.4820518587s: None
Nov 15 19:52:02 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 7de447e9-0f08-4dd7-9c9d-ca5409f0f46b (15)
Nov 15 19:52:22 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:22.106769
Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.345548
Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.345534
Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.345557
Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.346006
Nov 15 19:52:23 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:23.346028
Nov 15 19:52:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:45.787075
Nov 15 19:52:45 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:45.787217
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132246
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132708
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132727
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132862
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.132895
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.133633
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 7de447e9-0f08-4dd7-9c9d-ca5409f0f46b.
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:46.403596
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:51:14.765000+00:00 / 2017-11-16 00:51:19.801492+00:00
Nov 15 19:52:46 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:52:46 sat-r220-02 pulp: py.warnings:WARNING: (12800-85088) /usr/lib/python2.7/site-packages/pulp/server/managers/schedule/utils.py:118: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Nov 15 19:52:46 sat-r220-02 pulp: py.warnings:WARNING: (12800-85088) ScheduledCall.get_collection().remove({'resource': resource})
Nov 15 19:52:46 sat-r220-02 pulp: py.warnings:WARNING: (12800-85088)
Nov 15 19:52:49 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c50a2d83-289f-4bee-941e-d56a77717672]
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462142
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462155
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462208
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462178
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462861
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.462980
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.463809
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.463898
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.463873
Nov 15 19:52:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:52:59.465081
Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.control:INFO: Terminating f01a25e8-5fb1-419d-9b01-17d494d6d9f8 (15)
Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Task pulp.server.tasks.repository.delete[f01a25e8-5fb1-419d-9b01-17d494d6d9f8] raised unexpected: Terminated(15,)
Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Traceback (most recent call last):
Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated
Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) raise Terminated(-(signum or 0))
Nov 15 19:53:12 sat-r220-02 pulp: celery.worker.job:ERROR: (28132-85088) Terminated: 15
Nov 15 19:53:12 sat-r220-02 pulp: py.warnings:WARNING: [541dab26] (12968-85088) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Nov 15 19:53:12 sat-r220-02 pulp: py.warnings:WARNING: [541dab26] (12968-85088) "MongoClient opened before fork. Create MongoClient "
Nov 15 19:53:12 sat-r220-02 pulp: py.warnings:WARNING: [541dab26] (12968-85088)
Nov 15 19:53:20 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:20.826412
Nov 15 19:53:20 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:20.827871
Nov 15 19:53:25 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:25.978110
Nov 15 19:53:26 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: f01a25e8-5fb1-419d-9b01-17d494d6d9f8.
Nov 15 19:53:26 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:26.397730
Nov 15 19:53:26 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c50a2d83-289f-4bee-941e-d56a77717672] succeeded in 37.077276946s: None
Nov 15 19:53:27 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:27.992563
Nov 15 19:53:32 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:32.846345
Nov 15 19:53:35 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker scheduler@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.462000+00:00 / 2017-11-16 00:53:06.339549+00:00
Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.465000+00:00 / 2017-11-16 00:53:06.339549+00:00
Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.287075
Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.287722
Nov 15 19:53:36 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[541dab26-2ffc-4b66-8d2d-1c494559d1ca] succeeded in 23.6107569775s: None
Nov 15 19:53:36 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[ace87344-b537-4f41-be02-2df46bc26af5]
Nov 15 19:53:36 sat-r220-02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[39445f3f-39cd-47fb-a9c2-84c9ca713a9a]
Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.425234
Nov 15 19:53:36 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:36.479119
Nov 15 19:53:47 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:47.349187
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.315762
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.368979
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.369100
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'scheduler@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.769618
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.770920
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.463000+00:00 / 2017-11-16 00:53:06.339549+00:00
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:52.771069
Nov 15 19:53:52 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:53:54 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 53f6365c-7aaf-4cdb-a28b-af988c6d22b9 (15)
Nov 15 19:53:58 sat-r220-02 puppet-master[29050]: Compiled catalog for sat-r220-02.lab.eng.rdu2.redhat.com in environment production in 0.01 seconds
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 53f6365c-7aaf-4cdb-a28b-af988c6d22b9.
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-1@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.745846
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-4@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.745948
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' discovered
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.747478
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-0@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.747467
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-2@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:58.748681
Nov 15 19:53:58 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: ced18cf7-e816-4abf-a365-bacb1cdda736.
Nov 15 19:53:58 sat-r220-02 pulp: celery.worker.control:INFO: Terminating ced18cf7-e816-4abf-a365-bacb1cdda736 (15)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-7@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.029131
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 30073c56-cf84-482d-b279-7eb0ebe82ba6 (15)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 30073c56-cf84-482d-b279-7eb0ebe82ba6.
Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: (12968-85088) /usr/lib/python2.7/site-packages/pulp/server/managers/schedule/utils.py:118: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: (12968-85088) ScheduledCall.get_collection().remove({'resource': resource})
Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: (12968-85088)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.360683
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'reserved_resource_worker-6@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.361567
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 4cc2dc28-af1b-4922-9307-881368dbb3dd (15)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 4cc2dc28-af1b-4922-9307-881368dbb3dd.
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating d188c7cc-5050-466c-93b8-b0d207472567 (15)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.worker_watcher:INFO: Worker heartbeat from 'resource_manager@sat-r220-02.lab.eng.rdu2.redhat.com' at time 2017-11-16 00:53:59.754305
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: d188c7cc-5050-466c-93b8-b0d207472567.
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 91b0c678-85d5-4b07-93ac-8276496016aa (15)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 91b0c678-85d5-4b07-93ac-8276496016aa.
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.control:INFO: Terminating 0cde6a92-765e-4219-9365-c4cc18436b0a (15)
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) Task pulp.server.managers.repo.unit_association.unassociate_by_criteria[0cde6a92-765e-4219-9365-c4cc18436b0a] raised unexpected: Terminated(15,)
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) Traceback (most recent call last):
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1674, in _set_terminated
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) raise Terminated(-(signum or 0))
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:ERROR: (28129-43872) Terminated: 15
Nov 15 19:53:59 sat-r220-02 celery: reserved_resource_worker-5@sat-r220-02.lab.eng.rdu2.redhat.com ready.
Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: [8500f39b] (13272-43872) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: [8500f39b] (13272-43872) "MongoClient opened before fork. Create MongoClient "
Nov 15 19:53:59 sat-r220-02 pulp: py.warnings:WARNING: [8500f39b] (13272-43872)
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:INFO: Task canceled: 0cde6a92-765e-4219-9365-c4cc18436b0a.
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: Worker reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com has gone missing. Heartbeat: 2017-11-16 00:52:59.462000+00:00 / 2017-11-16 00:53:06.339549+00:00
Nov 15 19:53:59 sat-r220-02 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@sat-r220-02.lab.eng.rdu2.redhat.com is missing. Canceling the tasks in its queue.
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[4cc2dc28-af1b-4922-9307-881368dbb3dd]
Nov 15 19:53:59 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8500f39b-f9b1-4039-a2b3-6e745fc9dbcb] succeeded in 0.12942882441s: None
Nov 15 19:54:00 sat-r220-02 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[30073c56-cf84-482d-b279-7eb0ebe82ba6]
Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[2b78a57a-a87a-4d83-b44c-f70c1158e8ce] succeeded in 0.0994412815198s: None
Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.unit_association.unassociate_by_criteria[53f6365c-7aaf-4cdb-a28b-af988c6d22b9]
Nov 15 19:54:00 sat-r220-02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b2660f0c-43e1-4d30-a7ca-ae3575db68af] succeeded in 0.275608449243s: None
Nov 15 19:54:00 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:542: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Nov 15 19:54:00 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) RepoSyncResult.get_collection().remove({'repo_id': repo_id})
Nov 15 19:54:00 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088)
Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:543: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088) RepoPublishResult.get_collection().remove({'repo_id': repo_id})
Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [ace87344] (12968-85088)
Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [36c8cd54] (13272-43872) /usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [36c8cd54] (13272-43872) upsert=upsert, **write_concern)
Nov 15 19:54:02 sat-r220-02 pulp: py.warnings:WARNING: [36c8cd54] (13272-43872)
(2-2/2)