***** SERVER RESTARTED ***** Thu Sep 26 10:12:11.904 [mongosMain] MongoS version 2.4.4 starting: pid=21149 port=27017 64-bit host=lxrsvudbdt001 (--help for usage) Thu Sep 26 10:12:11.904 [mongosMain] git version: 4ec1fb96702c9d4c57b1e06dd34eb73a16e407d2 modules: subscription Thu Sep 26 10:12:11.904 [mongosMain] build info: Linux bs-e-rhel57 2.6.18-274.el5xen #1 SMP Fri Jul 8 17:45:44 EDT 2011 x86_64 BOOST_LIB_VERSION=1_49 Thu Sep 26 10:12:11.904 [mongosMain] options: { configdb: "lxrsvnsqlt01:27018", keyFile: "/udb/udbll002/data/m_data/key_file/keyfile.conf", logappend: true, logpath: "/udb/udbll002/data/m_data/mongos/mongos.log", port: 27017, upgrade: true } Thu Sep 26 10:12:11.928 [LockPinger] creating distributed lock ping thread for lxrsvnsqlt01:27018 and process lxrsvudbdt001:27017:1380204731:1804289383 (sleeping for 30000ms) Thu Sep 26 10:12:11.937 [mongosMain] distributed lock 'configUpgrade/lxrsvudbdt001:27017:1380204731:1804289383' acquired, ts : 524440bbb32b118fa7674567 Thu Sep 26 10:12:11.938 [mongosMain] starting upgrade of config server from v3 to v4 Thu Sep 26 10:12:11.938 [mongosMain] starting next upgrade step from v3 to v4 Thu Sep 26 10:12:11.938 [mongosMain] about to log new metadata event: { _id: "lxrsvudbdt001-2013-09-26T14:12:11-524440bbb32b118fa7674568", server: "lxrsvudbdt001", clientAddr: "N/A", time: new Date(1380204731938), what: "starting upgrade of config database", ns: "config.version", details: { from: 3, to: 4 } } Thu Sep 26 10:12:11.944 [mongosMain] stale mongos detected 1428 minutes ago, network location is lxrsvnsqlt01.metlife.com:27017, not checking version Thu Sep 26 10:12:11.944 [mongosMain] stale mongos detected 1429 minutes ago, network location is lxscvnsqlt01.metlife.com:27017, not checking version Thu Sep 26 10:12:11.945 [mongosMain] checking that version of host lxrsvnsqlt01:27019 is compatible with 2.2 Thu Sep 26 10:12:11.951 [mongosMain] checking that version of host lxscvnsqlt01:27019 is compatible with 2.2 Thu Sep 26 10:12:11.985 [mongosMain] checking that version of host lxrsvnsqlt01:27020 is compatible with 2.2 Thu Sep 26 10:12:11.996 [mongosMain] checking that version of host lxscvnsqlt01:27020 is compatible with 2.2 Thu Sep 26 10:12:12.032 [mongosMain] acquiring locks for 0 sharded collections... Thu Sep 26 10:12:12.032 [mongosMain] copying collection and chunk metadata to working and backup collections... Thu Sep 26 10:12:12.036 [mongosMain] About to copy config.collections to config.collections-upgrade-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.037 [mongosMain] Finished copying 0 documents from config.collections to config.collections-upgrade-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.045 [mongosMain] About to copy config.collections to config.collections-backup-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.046 [mongosMain] Finished copying 0 documents from config.collections to config.collections-backup-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.061 [mongosMain] About to copy config.chunks to config.chunks-upgrade-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.062 [mongosMain] Finished copying 0 documents from config.chunks to config.chunks-upgrade-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.081 [mongosMain] About to copy config.chunks to config.chunks-backup-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.081 [mongosMain] Finished copying 0 documents from config.chunks to config.chunks-backup-524440bcb32b118fa7674569 Thu Sep 26 10:12:12.092 [mongosMain] entered critical section for config upgrade Thu Sep 26 10:12:12.107 [mongosMain] writing new version info and clusterId 524440bcb32b118fa767456a... Thu Sep 26 10:12:12.110 [mongosMain] about to log new metadata event: { _id: "lxrsvudbdt001-2013-09-26T14:12:12-524440bcb32b118fa767456b", server: "lxrsvudbdt001", clientAddr: "N/A", time: new Date(1380204732109), what: "finished upgrade of config database", ns: "config.version", details: { from: 3, to: 4 } } Thu Sep 26 10:12:12.110 [mongosMain] upgrade of config server to v4 successful Thu Sep 26 10:12:12.112 [mongosMain] distributed lock 'configUpgrade/lxrsvudbdt001:27017:1380204731:1804289383' unlocked. Thu Sep 26 10:12:12.114 [mongosMain] couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping Thu Sep 26 10:12:12.114 [mongosMain] waiting for connections on port 27017 Thu Sep 26 10:12:12.115 [websvr] admin web console waiting for connections on port 28017 Thu Sep 26 10:12:12.115 [Balancer] about to contact config servers and shards Thu Sep 26 10:12:12.116 [Balancer] starting new replica set monitor for replica set Shard1 with seed of lxrsvnsqlt01:27019,lxscvnsqlt01:27019 Thu Sep 26 10:12:12.118 [Balancer] successfully connected to seed lxrsvnsqlt01:27019 for replica set Shard1 Thu Sep 26 10:12:12.119 [Balancer] changing hosts to { 0: "lxrsvnsqlt01:27019", 1: "lxscvnsqlt01:27019" } from Shard1/ Thu Sep 26 10:12:12.119 [Balancer] trying to add new host lxrsvnsqlt01:27019 to replica set Shard1 Thu Sep 26 10:12:12.120 [Balancer] successfully connected to new host lxrsvnsqlt01:27019 in replica set Shard1 Thu Sep 26 10:12:12.120 [Balancer] trying to add new host lxscvnsqlt01:27019 to replica set Shard1 Thu Sep 26 10:12:12.129 [Balancer] successfully connected to new host lxscvnsqlt01:27019 in replica set Shard1 Thu Sep 26 10:12:12.135 [Balancer] Primary for replica set Shard1 changed to lxrsvnsqlt01:27019 Thu Sep 26 10:12:12.174 [Balancer] replica set monitor for replica set Shard1 started, address is Shard1/lxrsvnsqlt01:27019,lxscvnsqlt01:27019 Thu Sep 26 10:12:12.175 [ReplicaSetMonitorWatcher] starting Thu Sep 26 10:12:12.179 [Balancer] starting new replica set monitor for replica set Shard2 with seed of lxrsvnsqlt01:27020,lxscvnsqlt01:27020 Thu Sep 26 10:12:12.193 [Balancer] successfully connected to seed lxrsvnsqlt01:27020 for replica set Shard2 Thu Sep 26 10:12:12.197 [Balancer] changing hosts to { 0: "lxrsvnsqlt01:27020", 1: "lxscvnsqlt01:27020" } from Shard2/ Thu Sep 26 10:12:12.197 [Balancer] trying to add new host lxrsvnsqlt01:27020 to replica set Shard2 Thu Sep 26 10:12:12.198 [Balancer] successfully connected to new host lxrsvnsqlt01:27020 in replica set Shard2 Thu Sep 26 10:12:12.198 [Balancer] trying to add new host lxscvnsqlt01:27020 to replica set Shard2 Thu Sep 26 10:12:12.208 [Balancer] successfully connected to new host lxscvnsqlt01:27020 in replica set Shard2 Thu Sep 26 10:12:12.215 [Balancer] Primary for replica set Shard2 changed to lxrsvnsqlt01:27020 Thu Sep 26 10:12:12.262 [Balancer] replica set monitor for replica set Shard2 started, address is Shard2/lxrsvnsqlt01:27020,lxscvnsqlt01:27020 Thu Sep 26 10:12:12.267 [Balancer] config servers and shards contacted successfully Thu Sep 26 10:12:12.268 [Balancer] balancer id: lxrsvudbdt001:27017 started at Sep 26 10:12:12 Thu Sep 26 10:16:41.987 [LockPinger] cluster lxrsvnsqlt01:27018 pinged successfully at Thu Sep 26 10:16:41 2013 by distributed lock pinger 'lxrsvnsqlt01:27018/lxrsvudbdt001:27017:1380204731:1804289383', sleeping for 30000ms Thu Sep 26 10:21:42.043 [LockPinger] cluster lxrsvnsqlt01:27018 pinged successfully at Thu Sep 26 10:21:42 2013 by distributed lock pinger 'lxrsvnsqlt01:27018/lxrsvudbdt001:27017:1380204731:1804289383', sleeping for 30000ms Thu Sep 26 10:26:42.099 [LockPinger] cluster lxrsvnsqlt01:27018 pinged successfully at Thu Sep 26 10:26:42 2013 by distributed lock pinger 'lxrsvnsqlt01:27018/lxrsvudbdt001:27017:1380204731:1804289383', sleeping for 30000ms