Tue Sep 24 08:39:55.549 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:39:55.549 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:40:01.917 [Balancer] warning: distributed lock 'balancer/qns01:57720:1380004424:1804289383 did not propagate properly. :: caused by :: 8017 update not consistent  ns: config.locks query: { _id: "balancer", state: 0, ts: ObjectId('5241a4410758cc211350fb19') } update: { $set: { state: 1, who: "qns01:57720:1380004424:1804289383:Balancer:846930886", process: "qns01:57720:1380004424:1804289383", when: new Date(1380033601562), why: "doing balance round", ts: ObjectId('5241a4418f04cbfe5c47f5ef') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1380033558000|12, connectionId: 17188, waited: 54, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1380033602000|5, connectionId: 17317, waited: 12, err: null, ok: 1.0 }
Tue Sep 24 08:40:01.918 [Balancer] lock update won, completing lock propagation for 'balancer/qns01:57720:1380004424:1804289383'
Tue Sep 24 08:40:02.138 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a4418f04cbfe5c47f5ef
Tue Sep 24 08:40:02.359 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:40:26.793 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a45a8f04cbfe5c47f5f0
Tue Sep 24 08:40:27.020 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:40:33.471 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a4618f04cbfe5c47f5f1
Tue Sep 24 08:40:33.692 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:40:46.150 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a46d8f04cbfe5c47f5f2
Tue Sep 24 08:40:46.335 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:40:52.880 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a4748f04cbfe5c47f5f3
Tue Sep 24 08:40:53.065 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:40:59.354 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a47b8f04cbfe5c47f5f4
Tue Sep 24 08:40:59.833 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:41:12.841 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' acquired, ts : 5241a4878f04cbfe5c47f5f5
Tue Sep 24 08:41:13.368 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' unlocked. 
Tue Sep 24 08:41:37.787 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:41:37.787 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:42:02.201 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:42:02.201 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:42:08.499 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:42:08.499 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:42:20.946 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:42:20.946 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:42:33.295 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:42:33.295 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:42:39.554 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:42:39.554 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:42:45.556 [Balancer] DBClientCursor::init call() failed
Tue Sep 24 08:42:45.556 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 no data
Tue Sep 24 08:42:45.556 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:45.557 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:45.557 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:45.557 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:45.559 [Balancer] DBClientCursor::init call() failed
Tue Sep 24 08:42:45.559 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 no data
Tue Sep 24 08:42:45.559 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:45.559 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:45.644 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:42:45.644 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [6] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:42:51.646 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.646 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.646 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.647 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.656 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.657 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.657 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.658 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:51.799 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:42:51.799 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:42:54.548 [LockPinger] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:54.548 [LockPinger] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:54.550 [LockPinger] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:42:54.550 [LockPinger] warning: distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [6] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:42:57.801 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:42:57.801 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.801 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:42:57.802 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:42:57.802 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.802 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.803 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:42:57.803 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.803 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:42:57.803 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:42:57.804 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.804 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.804 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.819 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.820 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.821 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.822 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.822 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:42:57.962 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:42:57.962 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:42:59.963 [CheckConfigServers] DBClientCursor::init call() failed
Tue Sep 24 08:42:59.963 [CheckConfigServers] warning:  couldn't check on config server:pcrfclient01-prim-site-1:47720 ok for now : 10276 DBClientBase::findN: transport error: pcrfclient01-prim-site-1:47720 ns: admin.$cmd query: { getlasterror: 1 }
Tue Sep 24 08:43:03.963 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.963 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.963 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:03.964 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.964 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:03.964 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:03.965 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.965 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.965 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.966 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.967 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.968 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.968 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:03.968 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:04.064 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:04.065 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:10.066 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.066 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.067 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:10.067 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.067 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:10.067 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:10.068 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.068 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.068 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.069 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.069 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.071 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.072 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.072 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:10.384 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:10.384 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:16.386 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.387 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.387 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:16.387 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.387 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:16.387 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:16.388 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.388 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.388 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.390 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.391 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.393 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.393 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.393 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:16.527 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:16.527 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:22.529 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.530 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.530 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:22.530 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.530 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:22.531 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:22.531 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.532 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.532 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.584 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.585 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.586 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.587 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.587 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:22.707 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:22.707 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:24.551 [LockPinger] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:24.552 [LockPinger] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:24.553 [LockPinger] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:24.553 [LockPinger] warning: distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [6] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:28.708 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:28.709 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.709 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:28.709 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:28.709 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.710 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.710 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:28.711 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.711 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:28.711 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:28.711 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.712 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.712 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.717 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.759 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.761 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.761 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.761 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:28.923 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:28.923 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:34.923 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.924 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.924 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:34.924 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.924 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:34.925 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:34.925 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.926 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.926 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.926 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.926 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.928 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.928 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:34.928 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:35.246 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:35.246 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:41.246 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.247 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.247 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:41.247 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.248 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:41.248 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:41.248 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.249 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.249 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.250 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.250 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.253 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.253 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.253 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:41.845 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:41.845 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:47.845 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.846 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.846 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:47.846 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.846 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:47.847 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:47.847 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.848 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.848 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.850 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.850 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.852 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.852 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.852 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:47.972 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:47.972 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:53.973 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.974 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.974 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:43:53.975 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.975 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:43:53.975 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:43:53.976 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.976 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.976 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.977 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.977 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.980 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.980 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:53.980 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:43:54.104 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:54.104 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:54.554 [LockPinger] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:43:54.554 [LockPinger] warning: distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:43:59.988 [CheckConfigServers] warning:  couldn't check on config server:pcrfclient01-prim-site-1:47720 ok for now : 11002 socket exception [6] server [pcrfclient01-prim-site-1:47720] mongos connectionpool error: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.105 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:00.105 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.105 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:00.105 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:00.106 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.106 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.106 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:00.107 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.107 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:00.107 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:00.108 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.108 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.108 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.238 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.239 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.240 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.241 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.241 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:00.483 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:44:00.483 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:44:06.484 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.484 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.484 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:06.485 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.485 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:06.485 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:06.486 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.486 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.486 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.497 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.497 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.498 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.499 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.499 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:06.655 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:44:06.655 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:44:12.656 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.656 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.656 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:12.657 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.657 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:12.657 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:12.658 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.658 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.658 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.659 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.660 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [FAILED_STATE] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.661 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.662 [Balancer] reconnect pcrfclient01-prim-site-1:47720 failed couldn't connect to server pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.662 [Balancer] query failed to: pcrfclient01-prim-site-1:47720 exception: socket exception [CONNECT_ERROR] for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:12.913 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:44:12.913 [Balancer] caught exception while doing balance: exception creating distributed lock balancer/qns01:57720:1380004424:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-1:47720]  pcrfclient01-prim-site-1:47720:{}
Tue Sep 24 08:44:18.914 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:18.914 [Balancer] reconnect pcrfclient01-prim-site-1:47720 ok
Tue Sep 24 08:44:18.914 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:18.915 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:18.915 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:18.919 [Balancer] trying reconnect to pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:18.919 [Balancer] reconnect pcrfclient01-prim-site-1:47720 ok
Tue Sep 24 08:44:19.297 [Balancer] could not acquire lock 'balancer/qns01:57720:1380004424:1804289383' (another update won)
Tue Sep 24 08:44:19.297 [Balancer] distributed lock 'balancer/qns01:57720:1380004424:1804289383' was not acquired.
Tue Sep 24 08:44:25.302 [Balancer] DBClientCursor::init call() failed
Tue Sep 24 08:44:25.302 [Balancer] Detected bad connection created at 1380004424425674 microSec, clearing pool for pcrfclient01-prim-site-1:47720
Tue Sep 24 08:44:25.302 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:44:25.302 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:44:25.302 [Balancer] caught exception while doing balance: DBClientBase::findN: transport error: pcrfclient01-prim-site-1:47720 ns: admin.$cmd query: { serverStatus: 1 }
Tue Sep 24 08:44:31.303 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:31.304 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:31.304 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:31.307 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:31.307 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:31.307 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:44:55.327 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:44:55.327 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:44:55.328 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:45:56.267 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 08:45:55 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383', sleeping for 30000ms
Tue Sep 24 08:46:07.388 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.103:47720
Tue Sep 24 08:46:13.389 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:13.389 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:19.394 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:19.395 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:25.398 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:25.399 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:26.269 [LockPinger] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:46:26.269 [LockPinger] warning: distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-2:47720]  pcrfclient01-prim-site-2:47720:{}
Tue Sep 24 08:46:31.407 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.103:47720
Tue Sep 24 08:46:37.408 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:37.409 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:43.413 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:43.413 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:49.420 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.103:47720
Tue Sep 24 08:46:55.421 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:55.421 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:46:56.271 [LockPinger] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:46:56.271 [LockPinger] warning: distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [5] server [pcrfclient01-prim-site-2:47720]  pcrfclient01-prim-site-2:47720:{}
Tue Sep 24 08:47:00.061 [CheckConfigServers] DBClientCursor::init call() failed
Tue Sep 24 08:47:00.061 [CheckConfigServers] warning:  couldn't check on config server:pcrfclient01-prim-site-2:47720 ok for now : 10276 DBClientBase::findN: transport error: pcrfclient01-prim-site-2:47720 ns: admin.$cmd query: { getlasterror: 1 }
Tue Sep 24 08:47:01.425 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:47:01.426 [Balancer] reconnect pcrfclient01-prim-site-2:47720 failed couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:47:01.426 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:47:01.426 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:47:01.426 [Balancer] SyncClusterConnection connect fail to: pcrfclient01-prim-site-2:47720 errmsg: couldn't connect to server pcrfclient01-prim-site-2:47720
Tue Sep 24 08:47:01.426 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:47:07.432 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:47:07.432 [Balancer] reconnect pcrfclient01-prim-site-2:47720 ok
Tue Sep 24 08:47:49.517 [Balancer] trying reconnect to pcrfclient01-prim-site-2:47720
Tue Sep 24 08:47:49.517 [Balancer] reconnect pcrfclient01-prim-site-2:47720 ok
Tue Sep 24 08:49:00.114 [CheckConfigServers] DBClientCursor::init call() failed
Tue Sep 24 08:49:00.114 [CheckConfigServers] warning:  couldn't check on config server:pcrfclient02-prim-site-1:47720 ok for now : 10276 DBClientBase::findN: transport error: pcrfclient02-prim-site-1:47720 ns: admin.$cmd query: { getlasterror: 1 }
Tue Sep 24 08:49:01.574 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.204:47720
Tue Sep 24 08:49:07.575 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:07.575 [Balancer] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:13.579 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:13.580 [Balancer] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:19.583 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:19.584 [Balancer] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:25.588 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:25.589 [Balancer] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:28.729 [LockPinger] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:28.730 [LockPinger] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:28.730 [LockPinger] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:49:28.730 [LockPinger] warning: distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [6] server [pcrfclient02-prim-site-1:47720]  pcrfclient02-prim-site-1:47720:{}
Tue Sep 24 08:49:31.594 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:49:31.594 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:49:31.595 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:49:31.595 [Balancer] SyncClusterConnection connect fail to: pcrfclient02-prim-site-1:47720 errmsg: couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:31.599 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.204:47720
Tue Sep 24 08:49:37.600 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:37.600 [Balancer] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:43.606 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:43.606 [Balancer] reconnect pcrfclient02-prim-site-1:47720 failed couldn't connect to server pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:49.611 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:49:49.612 [Balancer] reconnect pcrfclient02-prim-site-1:47720 ok
Tue Sep 24 08:50:49.652 [Balancer] trying reconnect to pcrfclient02-prim-site-1:47720
Tue Sep 24 08:50:49.652 [Balancer] reconnect pcrfclient02-prim-site-1:47720 ok
Tue Sep 24 08:52:01.701 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 08:52:01.701 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 08:52:01.702 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 08:52:33.244 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 08:52:32 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383', sleeping for 30000ms
Tue Sep 24 08:57:37.633 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 08:57:36 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383', sleeping for 30000ms
Tue Sep 24 08:59:32.057 [Balancer] forcing lock 'balancer/qnss01:57720:1380005121:1804289383' because elapsed time 900815 > takeover time 900000
Tue Sep 24 08:59:32.057 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.103:47720
Tue Sep 24 08:59:32.058 [Balancer] Detected bad connection created at 1380004424426855 microSec, clearing pool for pcrfclient01-prim-site-2:47720
Tue Sep 24 08:59:32.062 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 08:59:32.062 [Balancer] caught exception while doing balance: exception forcing distributed lock balancer/qnss01:57720:1380005121:1804289383 :: caused by :: socket exception [SEND_ERROR] for 192.168.209.103:47720
Tue Sep 24 09:02:43.133 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 09:02:42 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383', sleeping for 30000ms
Tue Sep 24 09:03:14.222 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 09:03:14.223 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 09:03:14.223 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 09:07:51.270 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 09:07:50 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383', sleeping for 30000ms
Tue Sep 24 09:12:59.101 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 09:12:58 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380004424:1804289383', sleeping for 30000ms
Tue Sep 24 09:14:38.858 [Balancer] forcing lock 'balancer/qnss01:57720:1380005121:1804289383' because elapsed time 900861 > takeover time 900000
Tue Sep 24 09:14:38.906 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.103:47720
Tue Sep 24 09:14:38.906 [Balancer] Socket say send() errno:32 Broken pipe 192.168.209.204:47720
Tue Sep 24 09:14:38.907 [Balancer] scoped connection to pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 not being returned to the pool
Tue Sep 24 09:14:38.907 [Balancer] caught exception while doing balance: exception forcing distributed lock balancer/qnss01:57720:1380005121:1804289383 :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [2] server [192.168.209.103:47720]  pcrfclient01-prim-site-2:47720:{}9001 socket exception [2] server [192.168.209.204:47720]  pcrfclient02-prim-site-1:47720:{}
Tue Sep 24 09:16:02.966 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 09:16:02.966 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 09:16:02.967 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 09:16:07.088 [mongosMain] dbexit: received signal 2 rc:0 received signal 2


***** SERVER RESTARTED *****


Tue Sep 24 09:16:31.621 [mongosMain] MongoS version 2.4.6 starting: pid=1739 port=57720 64-bit host=qns01 (--help for usage)
Tue Sep 24 09:16:31.621 [mongosMain] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Tue Sep 24 09:16:31.621 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Tue Sep 24 09:16:31.621 [mongosMain] options: { configdb: "pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720", fork: true, logappend: true, logpath: "/var/log/mongos-57720.log", pidfilepath: "/var/run/mongos-57720.pid", port: 57720, quiet: true }
Tue Sep 24 09:16:31.627 [mongosMain] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 09:16:31.628 [mongosMain] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 09:16:31.628 [mongosMain] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 09:16:31.629 [mongosMain] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 09:16:31.630 [mongosMain] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 09:16:31.630 [mongosMain] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 09:16:32.090 [mongosMain] waiting for connections on port 57720
Tue Sep 24 09:16:32.090 [websvr] admin web console waiting for connections on port 58720
Tue Sep 24 09:16:32.090 [Balancer] about to contact config servers and shards
Tue Sep 24 09:16:32.091 [Balancer] starting new replica set monitor for replica set set04 with seed of sessionmgr01-site-1:27720,sessionmgr01-site-2:27720,sessionmgr02-site-1:27720,sessionmgr02-site-2:27720
Tue Sep 24 09:16:32.091 [Balancer] successfully connected to seed sessionmgr01-site-1:27720 for replica set set04
Tue Sep 24 09:16:32.092 [Balancer] changing hosts to { 0: "sessionmgr01-site-1:27720", 1: "sessionmgr02-site-2:27720", 2: "sessionmgr01-site-2:27720", 3: "sessionmgr02-site-1:27720" } from set04/
Tue Sep 24 09:16:32.092 [Balancer] trying to add new host sessionmgr01-site-1:27720 to replica set set04
Tue Sep 24 09:16:32.092 [Balancer] successfully connected to new host sessionmgr01-site-1:27720 in replica set set04
Tue Sep 24 09:16:32.092 [Balancer] trying to add new host sessionmgr01-site-2:27720 to replica set set04
Tue Sep 24 09:16:32.092 [Balancer] successfully connected to new host sessionmgr01-site-2:27720 in replica set set04
Tue Sep 24 09:16:32.092 [Balancer] trying to add new host sessionmgr02-site-1:27720 to replica set set04
Tue Sep 24 09:16:32.092 [Balancer] successfully connected to new host sessionmgr02-site-1:27720 in replica set set04
Tue Sep 24 09:16:32.092 [Balancer] trying to add new host sessionmgr02-site-2:27720 to replica set set04
Tue Sep 24 09:16:32.093 [Balancer] successfully connected to new host sessionmgr02-site-2:27720 in replica set set04
Tue Sep 24 09:16:32.245 [Balancer] Primary for replica set set04 changed to sessionmgr01-site-1:27720
Tue Sep 24 09:16:32.248 [Balancer] replica set monitor for replica set set04 started, address is set04/sessionmgr01-site-1:27720,sessionmgr01-site-2:27720,sessionmgr02-site-1:27720,sessionmgr02-site-2:27720
Tue Sep 24 09:16:32.248 [ReplicaSetMonitorWatcher] starting
Tue Sep 24 09:16:32.249 [Balancer] starting new replica set monitor for replica set set04a with seed of sessionmgr03-site-1:27720,sessionmgr03-site-2:27720,sessionmgr04-site-1:27720,sessionmgr04-site-2:27720
Tue Sep 24 09:16:32.249 [Balancer] successfully connected to seed sessionmgr03-site-1:27720 for replica set set04a
Tue Sep 24 09:16:32.250 [Balancer] changing hosts to { 0: "sessionmgr03-site-1:27720", 1: "sessionmgr04-site-2:27720", 2: "sessionmgr03-site-2:27720", 3: "sessionmgr04-site-1:27720" } from set04a/
Tue Sep 24 09:16:32.250 [Balancer] trying to add new host sessionmgr03-site-1:27720 to replica set set04a
Tue Sep 24 09:16:32.250 [Balancer] successfully connected to new host sessionmgr03-site-1:27720 in replica set set04a
Tue Sep 24 09:16:32.250 [Balancer] trying to add new host sessionmgr03-site-2:27720 to replica set set04a
Tue Sep 24 09:16:32.250 [Balancer] successfully connected to new host sessionmgr03-site-2:27720 in replica set set04a
Tue Sep 24 09:16:32.250 [Balancer] trying to add new host sessionmgr04-site-1:27720 to replica set set04a
Tue Sep 24 09:16:32.251 [Balancer] successfully connected to new host sessionmgr04-site-1:27720 in replica set set04a
Tue Sep 24 09:16:32.251 [Balancer] trying to add new host sessionmgr04-site-2:27720 to replica set set04a
Tue Sep 24 09:16:32.251 [Balancer] successfully connected to new host sessionmgr04-site-2:27720 in replica set set04a
Tue Sep 24 09:16:32.431 [Balancer] Primary for replica set set04a changed to sessionmgr04-site-1:27720
Tue Sep 24 09:16:32.433 [Balancer] replica set monitor for replica set set04a started, address is set04a/sessionmgr03-site-1:27720,sessionmgr03-site-2:27720,sessionmgr04-site-1:27720,sessionmgr04-site-2:27720
Tue Sep 24 09:16:32.434 [Balancer] starting new replica set monitor for replica set set04b with seed of sessionmgr05-site-1:27720,sessionmgr05-site-2:27720,sessionmgr06-site-1:27720,sessionmgr06-site-2:27720
Tue Sep 24 09:16:32.434 [Balancer] successfully connected to seed sessionmgr05-site-1:27720 for replica set set04b
Tue Sep 24 09:16:32.435 [Balancer] changing hosts to { 0: "sessionmgr05-site-1:27720", 1: "sessionmgr06-site-2:27720", 2: "sessionmgr05-site-2:27720", 3: "sessionmgr06-site-1:27720" } from set04b/
Tue Sep 24 09:16:32.435 [Balancer] trying to add new host sessionmgr05-site-1:27720 to replica set set04b
Tue Sep 24 09:16:32.435 [Balancer] successfully connected to new host sessionmgr05-site-1:27720 in replica set set04b
Tue Sep 24 09:16:32.435 [Balancer] trying to add new host sessionmgr05-site-2:27720 to replica set set04b
Tue Sep 24 09:16:32.435 [Balancer] successfully connected to new host sessionmgr05-site-2:27720 in replica set set04b
Tue Sep 24 09:16:32.435 [Balancer] trying to add new host sessionmgr06-site-1:27720 to replica set set04b
Tue Sep 24 09:16:32.436 [Balancer] successfully connected to new host sessionmgr06-site-1:27720 in replica set set04b
Tue Sep 24 09:16:32.436 [Balancer] trying to add new host sessionmgr06-site-2:27720 to replica set set04b
Tue Sep 24 09:16:32.436 [Balancer] successfully connected to new host sessionmgr06-site-2:27720 in replica set set04b
Tue Sep 24 09:16:32.647 [Balancer] Primary for replica set set04b changed to sessionmgr06-site-1:27720
Tue Sep 24 09:16:32.649 [Balancer] replica set monitor for replica set set04b started, address is set04b/sessionmgr05-site-1:27720,sessionmgr05-site-2:27720,sessionmgr06-site-1:27720,sessionmgr06-site-2:27720
Tue Sep 24 09:16:32.650 [Balancer] config servers and shards contacted successfully
Tue Sep 24 09:16:32.650 [Balancer] balancer id: qns01:57720 started at Sep 24 09:16:32
Tue Sep 24 09:16:32.650 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-1:47720]
Tue Sep 24 09:16:32.650 [Balancer] SyncClusterConnection connecting to [pcrfclient01-prim-site-2:47720]
Tue Sep 24 09:16:32.650 [Balancer] SyncClusterConnection connecting to [pcrfclient02-prim-site-1:47720]
Tue Sep 24 09:16:32.659 [LockPinger] creating distributed lock ping thread for pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 and process qns01:57720:1380035792:1804289383 (sleeping for 30000ms)
Tue Sep 24 09:21:05.048 [LockPinger] cluster pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720 pinged successfully at Tue Sep 24 09:21:04 2013 by distributed lock pinger 'pcrfclient01-prim-site-1:47720,pcrfclient01-prim-site-2:47720,pcrfclient02-prim-site-1:47720/qns01:57720:1380035792:1804289383', sleeping for 30000ms
Tue Sep 24 09:21:57.081 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae14b53137f5cbbdea9d
Tue Sep 24 09:21:57.241 [Balancer] ChunkManager: time to load chunks for spr.subscriber: 15ms sequenceNumber: 2 version: 464|19||523ae679032dee9f573fe69c based on: (empty)
Tue Sep 24 09:21:57.368 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:22:09.628 [Balancer] could not acquire lock 'balancer/qns01:57720:1380035792:1804289383' (another update won)
Tue Sep 24 09:22:09.628 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' was not acquired.
Tue Sep 24 09:22:21.780 [Balancer] could not acquire lock 'balancer/qns01:57720:1380035792:1804289383' (another update won)
Tue Sep 24 09:22:21.780 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' was not acquired.
Tue Sep 24 09:22:28.040 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae33b53137f5cbbdeaa0
Tue Sep 24 09:22:28.112 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:22:58.712 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae52b53137f5cbbdeaa1
Tue Sep 24 09:22:59.221 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:23:29.453 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae71b53137f5cbbdeaa2
Tue Sep 24 09:23:29.651 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:23:48.097 [conn4] creating WriteBackListener for: sessionmgr01-site-1:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.097 [conn4] creating WriteBackListener for: sessionmgr01-site-2:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.097 [conn4] creating WriteBackListener for: sessionmgr02-site-1:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.097 [conn4] creating WriteBackListener for: sessionmgr02-site-2:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.099 [conn4] creating WriteBackListener for: sessionmgr03-site-1:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.099 [conn4] creating WriteBackListener for: sessionmgr03-site-2:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.099 [conn4] creating WriteBackListener for: sessionmgr04-site-1:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.099 [conn4] creating WriteBackListener for: sessionmgr04-site-2:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.101 [conn4] creating WriteBackListener for: sessionmgr05-site-1:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.101 [conn4] creating WriteBackListener for: sessionmgr05-site-2:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.101 [conn4] creating WriteBackListener for: sessionmgr06-site-1:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.101 [conn4] creating WriteBackListener for: sessionmgr06-site-2:27720 serverID: 5241acd0b53137f5cbbdea9c
Tue Sep 24 09:23:48.282 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae83b53137f5cbbdeaa3
Tue Sep 24 09:23:48.427 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:23:54.742 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae8ab53137f5cbbdeaa4
Tue Sep 24 09:23:54.849 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:24:01.209 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae90b53137f5cbbdeaa5
Tue Sep 24 09:24:01.352 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:24:07.582 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae97b53137f5cbbdeaa6
Tue Sep 24 09:24:07.767 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:24:14.337 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241ae9db53137f5cbbdeaa7
Tue Sep 24 09:24:14.516 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:24:26.648 [Balancer] could not acquire lock 'balancer/qns01:57720:1380035792:1804289383' (another update won)
Tue Sep 24 09:24:26.648 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' was not acquired.
Tue Sep 24 09:24:33.205 [Balancer] could not acquire lock 'balancer/qns01:57720:1380035792:1804289383' (another update won)
Tue Sep 24 09:24:33.205 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' was not acquired.
Tue Sep 24 09:24:39.405 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241aeb7b53137f5cbbdeaaa
Tue Sep 24 09:24:39.476 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:24:51.840 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241aec3b53137f5cbbdeaab
Tue Sep 24 09:24:51.947 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:25:04.252 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' acquired, ts : 5241aecfb53137f5cbbdeaac
Tue Sep 24 09:25:04.431 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' unlocked. 
Tue Sep 24 09:25:34.581 [Balancer] could not acquire lock 'balancer/qns01:57720:1380035792:1804289383' (another update won)
Tue Sep 24 09:25:34.581 [Balancer] distributed lock 'balancer/qns01:57720:1380035792:1804289383' was not acquired.