Fri Aug 9 08:14:50.836 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:50.836 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.836 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:50.836 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4106 current: 4144 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c378e280 Fri Aug 9 08:14:50.836 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4144 Fri Aug 9 08:14:50.837 [conn279] setShardVersion success: { oldVersion: Timestamp 516000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:50.837 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.837 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.837 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:50.837 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.839 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:50.839 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.839 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.839 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:50.839 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.839 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:50.839 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.849 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:50.850 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:50.851 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:50.851 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 11ms sequenceNumber: 4145 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:50.854 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:50.854 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:50.854 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.854 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.854 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:50.854 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4144 current: 4145 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c8adb180 Fri Aug 9 08:14:50.854 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4145 Fri Aug 9 08:14:50.855 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:50.855 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.855 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.855 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:50.855 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.864 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:50.864 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:50.865 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:50.865 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 10ms sequenceNumber: 4146 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:50.867 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:50.869 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:50.869 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.869 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.869 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:50.869 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4145 current: 4146 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c7a3e280 Fri Aug 9 08:14:50.869 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4146 Fri Aug 9 08:14:50.870 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:50.870 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:50.870 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.870 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:50.870 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:50.871 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:51.873 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:51.873 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.873 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:51.873 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.873 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:51.873 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.875 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:51.875 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.875 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.875 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:51.875 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.875 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:51.875 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.886 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:51.887 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:51.888 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:51.888 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 12ms sequenceNumber: 4147 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:51.890 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:51.890 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:51.890 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.890 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.890 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:51.890 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4146 current: 4147 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c378e280 Fri Aug 9 08:14:51.890 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4147 Fri Aug 9 08:14:51.892 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:51.892 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.892 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.892 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:51.892 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.901 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:51.902 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:51.903 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:51.903 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 10ms sequenceNumber: 4148 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:51.905 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:51.906 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:51.906 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.906 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.906 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:51.906 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4147 current: 4148 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c2e44c80 Fri Aug 9 08:14:51.906 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4148 Fri Aug 9 08:14:51.907 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:51.907 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:51.907 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.907 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:51.907 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:51.908 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:52.909 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:52.909 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.909 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:52.909 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.909 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:52.909 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.911 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:52.911 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.911 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.911 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:52.911 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.911 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:52.911 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.921 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:52.922 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:52.923 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:52.923 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 11ms sequenceNumber: 4149 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:52.925 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:52.925 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:52.925 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.925 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.925 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:52.925 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4148 current: 4149 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c7a3e280 Fri Aug 9 08:14:52.925 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4149 Fri Aug 9 08:14:52.926 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:52.926 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.926 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.926 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:52.926 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.939 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:52.940 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:52.941 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:52.941 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 12ms sequenceNumber: 4150 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:52.943 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:52.944 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:52.944 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.944 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.944 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:52.944 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4149 current: 4150 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c3d6cf00 Fri Aug 9 08:14:52.944 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4150 Fri Aug 9 08:14:52.945 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:52.945 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:52.945 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.945 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:52.945 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:52.946 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:53.947 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:53.947 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.947 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:53.947 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.947 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:53.947 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.949 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:53.949 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.949 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.949 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:53.949 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.949 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:53.949 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.960 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:53.962 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:53.964 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:53.964 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 13ms sequenceNumber: 4151 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:53.967 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:53.967 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:53.967 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.967 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.967 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:53.967 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4150 current: 4151 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c2e44c80 Fri Aug 9 08:14:53.967 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4151 Fri Aug 9 08:14:53.968 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:53.969 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.969 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.969 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:53.969 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.980 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:53.980 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:53.982 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:53.983 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 13ms sequenceNumber: 4152 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:53.985 [conn254] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:53.985 [conn254] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.985 [conn254] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:53.985 [conn254] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4106 current: 4151 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c2e44c80 Fri Aug 9 08:14:53.985 [conn254] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16cf2c5ea0 4151 Fri Aug 9 08:14:53.986 [conn254] setShardVersion success: { oldVersion: Timestamp 516000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:53.986 [conn254] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.986 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:53.986 [conn254] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.986 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:53.986 [conn254] [pcursor] finishing over 1 shards Fri Aug 9 08:14:53.986 [conn254] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.986 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.986 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.986 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:53.986 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4151 current: 4152 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16cb6d3400 Fri Aug 9 08:14:53.986 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4152 Fri Aug 9 08:14:53.993 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:53.993 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:53.993 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.993 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:53.993 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:53.995 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:54.997 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:54.997 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:54.997 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:54.997 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:54.997 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:54.998 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.001 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:55.001 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.001 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.001 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:55.002 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.002 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:55.002 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.023 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:55.023 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:55.027 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:55.027 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 18ms sequenceNumber: 4153 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:55.029 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:55.029 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:55.029 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.029 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.029 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:55.029 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4152 current: 4153 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c3d6cf00 Fri Aug 9 08:14:55.029 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4153 Fri Aug 9 08:14:55.037 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:55.037 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.037 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.037 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:55.037 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.056 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:55.056 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:55.058 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:55.058 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 19ms sequenceNumber: 4154 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:55.060 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:55.062 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:55.062 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.062 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.062 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:55.062 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4153 current: 4154 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c8adb180 Fri Aug 9 08:14:55.062 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4154 Fri Aug 9 08:14:55.067 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:55.067 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:55.067 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.067 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:55.067 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:55.068 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:56.069 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:56.069 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.069 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:56.069 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.069 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:56.069 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.071 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:56.071 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.071 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.071 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:56.071 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.071 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:56.071 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.081 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:56.082 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:56.085 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:56.085 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 12ms sequenceNumber: 4155 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:56.087 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:56.087 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:56.087 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.087 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.087 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:56.087 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4154 current: 4155 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16cb6d3400 Fri Aug 9 08:14:56.087 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4155 Fri Aug 9 08:14:56.088 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:56.088 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.089 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.089 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:56.089 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.100 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:56.101 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:56.103 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:56.103 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 13ms sequenceNumber: 4156 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:56.106 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:56.108 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:56.108 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.108 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.108 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:56.109 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4155 current: 4156 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c3710280 Fri Aug 9 08:14:56.109 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4156 Fri Aug 9 08:14:56.112 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:56.112 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:56.112 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.112 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:56.112 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:56.113 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:56.225 [Balancer] Refreshing MaxChunkSize: 64 Fri Aug 9 08:14:56.227 [Balancer] trying to acquire new distributed lock for balancer on w1.fnd:27019,w2.fnd:27019,w3.fnd:27019 ( lock timeout : 900000, ping interval : 30000, process : w3.fnd:27017:1374781534:1804289383 ) Fri Aug 9 08:14:56.228 [Balancer] about to acquire distributed lock 'balancer/w3.fnd:27017:1374781534:1804289383: "when" : { "$date" : "Fri Aug 9 08:14:56 2013" }, Fri Aug 9 08:14:56.238 [ReplicaSetMonitorWatcher] checking replica set: db1 Fri Aug 9 08:14:56.239 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db1-m.fnd:27018 { setName: "db1", ismaster: true, secondary: false, hosts: [ "db1-m.fnd:27018", "db1-s.fnd:27018" ], arbiters: [ "w1.fnd:27020" ], primary: "db1-m.fnd:27018", me: "db1-m.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096241), ok: 1.0 } Fri Aug 9 08:14:56.240 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db1-s.fnd:27018 Fri Aug 9 08:14:56.240 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w1.fnd:27020) Fri Aug 9 08:14:56.240 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db1-m.fnd:27018 Fri Aug 9 08:14:56.240 [ReplicaSetMonitorWatcher] _check : db1/db1-m.fnd:27018,db1-s.fnd:27018 Fri Aug 9 08:14:56.240 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db1-m.fnd:27018 { setName: "db1", ismaster: true, secondary: false, hosts: [ "db1-m.fnd:27018", "db1-s.fnd:27018" ], arbiters: [ "w1.fnd:27020" ], primary: "db1-m.fnd:27018", me: "db1-m.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096242), ok: 1.0 } Fri Aug 9 08:14:56.241 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db1-s.fnd:27018 Fri Aug 9 08:14:56.241 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w1.fnd:27020) Fri Aug 9 08:14:56.241 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db1-m.fnd:27018 Fri Aug 9 08:14:56.242 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db1-m.fnd:27018 { setName: "db1", ismaster: true, secondary: false, hosts: [ "db1-m.fnd:27018", "db1-s.fnd:27018" ], arbiters: [ "w1.fnd:27020" ], primary: "db1-m.fnd:27018", me: "db1-m.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096244), ok: 1.0 } Fri Aug 9 08:14:56.243 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db1-s.fnd:27018 Fri Aug 9 08:14:56.243 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w1.fnd:27020) Fri Aug 9 08:14:56.243 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db1-m.fnd:27018 Fri Aug 9 08:14:56.244 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db1-s.fnd:27018 { setName: "db1", ismaster: false, secondary: true, hosts: [ "db1-s.fnd:27018", "db1-m.fnd:27018" ], arbiters: [ "w1.fnd:27020" ], primary: "db1-m.fnd:27018", me: "db1-s.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096244), ok: 1.0 } Fri Aug 9 08:14:56.245 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db1-s.fnd:27018 Fri Aug 9 08:14:56.245 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w1.fnd:27020) Fri Aug 9 08:14:56.245 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db1-m.fnd:27018 Fri Aug 9 08:14:56.245 [ReplicaSetMonitorWatcher] checking replica set: db2 Fri Aug 9 08:14:56.246 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db2-m.fnd:27018 { setName: "db2", ismaster: true, secondary: false, hosts: [ "db2-m.fnd:27018", "db2-s.fnd:27018" ], arbiters: [ "w2.fnd:27020" ], primary: "db2-m.fnd:27018", me: "db2-m.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096246), ok: 1.0 } Fri Aug 9 08:14:56.247 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db2-m.fnd:27018 Fri Aug 9 08:14:56.247 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db2-s.fnd:27018 Fri Aug 9 08:14:56.247 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w2.fnd:27020) Fri Aug 9 08:14:56.247 [ReplicaSetMonitorWatcher] _check : db2/db2-m.fnd:27018,db2-s.fnd:27018 Fri Aug 9 08:14:56.247 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db2-m.fnd:27018 { setName: "db2", ismaster: true, secondary: false, hosts: [ "db2-m.fnd:27018", "db2-s.fnd:27018" ], arbiters: [ "w2.fnd:27020" ], primary: "db2-m.fnd:27018", me: "db2-m.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096248), ok: 1.0 } Fri Aug 9 08:14:56.248 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db2-m.fnd:27018 Fri Aug 9 08:14:56.248 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db2-s.fnd:27018 Fri Aug 9 08:14:56.248 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w2.fnd:27020) Fri Aug 9 08:14:56.249 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db2-m.fnd:27018 { setName: "db2", ismaster: true, secondary: false, hosts: [ "db2-m.fnd:27018", "db2-s.fnd:27018" ], arbiters: [ "w2.fnd:27020" ], primary: "db2-m.fnd:27018", me: "db2-m.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096250), ok: 1.0 } Fri Aug 9 08:14:56.250 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db2-m.fnd:27018 Fri Aug 9 08:14:56.250 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db2-s.fnd:27018 Fri Aug 9 08:14:56.250 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w2.fnd:27020) Fri Aug 9 08:14:56.251 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: db2-s.fnd:27018 { setName: "db2", ismaster: false, secondary: true, hosts: [ "db2-s.fnd:27018", "db2-m.fnd:27018" ], arbiters: [ "w2.fnd:27020" ], primary: "db2-m.fnd:27018", me: "db2-s.fnd:27018", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1376036096252), ok: 1.0 } Fri Aug 9 08:14:56.251 [ReplicaSetMonitorWatcher] dbclient_rs nodes[0].ok = true db2-m.fnd:27018 Fri Aug 9 08:14:56.251 [ReplicaSetMonitorWatcher] dbclient_rs nodes[1].ok = true db2-s.fnd:27018 Fri Aug 9 08:14:56.251 [ReplicaSetMonitorWatcher] dbclient_rs _checkStatus couldn't _find(w2.fnd:27020) Fri Aug 9 08:14:56.390 [Balancer] distributed lock 'balancer/w3.fnd:27017:1374781534:1804289383' acquired, ts : 5204a5004d284be232ad238a Fri Aug 9 08:14:56.390 [Balancer] *** start balancing round Fri Aug 9 08:14:56.390 [Balancer] waitForDelete: 0 Fri Aug 9 08:14:56.390 [Balancer] secondaryThrottle: 1 Fri Aug 9 08:14:56.450 [Balancer] collection : live-db.RecoUser Fri Aug 9 08:14:56.451 [Balancer] donor : db1 chunks on 974 Fri Aug 9 08:14:56.451 [Balancer] receiver : db2 chunks on 973 Fri Aug 9 08:14:56.451 [Balancer] threshold : 8 Fri Aug 9 08:14:56.461 [Balancer] no need to move any chunk Fri Aug 9 08:14:56.461 [Balancer] *** end of balancing round Fri Aug 9 08:14:56.527 [Balancer] distributed lock 'balancer/w3.fnd:27017:1374781534:1804289383' unlocked. Fri Aug 9 08:14:57.114 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:57.114 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.114 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:57.115 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.115 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:57.115 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.116 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:57.116 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.116 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.116 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:57.117 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.117 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:57.117 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.128 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:57.128 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:57.129 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:57.129 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 11ms sequenceNumber: 4157 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:57.132 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:57.132 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:57.132 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.132 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.132 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:57.132 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4156 current: 4157 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c8adb180 Fri Aug 9 08:14:57.132 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4157 Fri Aug 9 08:14:57.133 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:57.133 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.133 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.134 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:57.134 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.143 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:57.144 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:57.145 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:57.145 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 10ms sequenceNumber: 4158 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:57.147 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:57.148 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:57.148 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.148 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.148 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:57.148 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4157 current: 4158 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c7a3ea00 Fri Aug 9 08:14:57.148 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4158 Fri Aug 9 08:14:57.149 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:57.149 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:57.149 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.149 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:57.149 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:57.150 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send ) Fri Aug 9 08:14:58.151 [conn279] [pcursor] creating pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:58.151 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.151 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:58.151 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.151 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:58.151 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.153 [conn279] [pcursor] retrying (1 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:58.153 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.153 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.153 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:58.153 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.153 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:58.153 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.163 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:58.164 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:58.165 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:58.165 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 11ms sequenceNumber: 4159 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:58.168 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:58.168 [conn279] [pcursor] retrying (2 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:58.168 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.168 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.168 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:58.168 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4158 current: 4159 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c8adb180 Fri Aug 9 08:14:58.168 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4159 Fri Aug 9 08:14:58.169 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:58.169 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.169 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.169 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:58.169 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.178 [conn279] loading chunk manager for collection live-db.RecoUser using old chunk manager w/ version 520|3||516fe84d12c6006490b9d37a and 1947 chunks Fri Aug 9 08:14:58.178 [conn279] major version query from 520|3||516fe84d12c6006490b9d37a and over 2 shards is { ns: "live-db.RecoUser", $or: [ { lastmod: { $gte: Timestamp 520000|3 } }, { shard: "db1", lastmod: { $gt: Timestamp 520000|3 } }, { shard: "db2", lastmod: { $gt: Timestamp 520000|0 } } ] } Fri Aug 9 08:14:58.179 [conn279] loaded 1 chunks into new chunk manager for live-db.RecoUser with version 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:58.179 [conn279] ChunkManager: time to load chunks for live-db.RecoUser: 10ms sequenceNumber: 4160 version: 520|3||516fe84d12c6006490b9d37a based on: 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:58.182 [conn279] warning: chunk manager reload forced for collection 'live-db.RecoUser', config version is 520|3||516fe84d12c6006490b9d37a Fri Aug 9 08:14:58.183 [conn279] [pcursor] retrying (3 tries) pcursor over QSpec { ns: "live-db.RecoUser", n2skip: 0, n2return: 0, options: 0, query: { $query: { _id: ObjectId('4f048c822cdcfe9116ab076b') }, $readPreference: { mode: "primary" } }, fields: {} } and CInfo { v_ns: "", filter: {} } Fri Aug 9 08:14:58.183 [conn279] [pcursor] closing cursor on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018 as the connection is no longer required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.183 [conn279] [pcursor] initializing over 1 shards required by [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.183 [conn279] [pcursor] initializing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: {}, retryNext: false, init: false, finish: false, errored: false } Fri Aug 9 08:14:58.183 [conn279] have to set shard version for conn: db2-m.fnd:27018 ns:live-db.RecoUser my last seq: 4159 current: 4160 version: 520|0||516fe84d12c6006490b9d37a manager: 0x7f16c7a3e500 Fri Aug 9 08:14:58.183 [conn279] setShardVersion db2 db2-m.fnd:27018 live-db.RecoUser { setShardVersion: "live-db.RecoUser", configdb: "w1.fnd:27019,w2.fnd:27019,w3.fnd:27019", version: Timestamp 520000|0, versionEpoch: ObjectId('516fe84d12c6006490b9d37a'), serverID: ObjectId('51f1805e4d284be232aaa457'), shard: "db2", shardHost: "db2/db2-m.fnd:27018,db2-s.fnd:27018" } 0x7f16c889f340 4160 Fri Aug 9 08:14:58.184 [conn279] setShardVersion success: { oldVersion: Timestamp 520000|0, oldVersionEpoch: ObjectId('516fe84d12c6006490b9d37a'), ok: 1.0 } Fri Aug 9 08:14:58.184 [conn279] [pcursor] needed to set remote version on connection to value compatible with [live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a] Fri Aug 9 08:14:58.184 [conn279] [pcursor] initialized query (lazily) on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.184 [conn279] [pcursor] finishing over 1 shards Fri Aug 9 08:14:58.184 [conn279] [pcursor] finishing on shard db2:db2/db2-m.fnd:27018,db2-s.fnd:27018, current connection state is { state: { conn: "db2/db2-m.fnd:27018,db2-s.fnd:27018", vinfo: "live-db.RecoUser @ 520|3||516fe84d12c6006490b9d37a", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false } Fri Aug 9 08:14:58.185 [conn279] AssertionException while processing op type : 2004 to : live-db.RecoUser :: caused by :: 13388 too many retries of stale version info ( ns : live-db.RecoUser, received : 516|0||516fe84d12c6006490b9d37a, wanted : 520|0||516fe84d12c6006490b9d37a, send )