Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-4453

Shards not picking up replica set updates

    • Linux

      When adding next replica to set we expect mongo to add this node to shard automatically, but in our setup it has never happened.

      e.g. after successfully adding the second replica to set we get rs config:

      PRIMARY> rs.status()
      {
      "set" : "rs-0",
      "date" : ISODate("2011-12-08T15:46:26Z"),
      "myState" : 1,
      "members" : [
      {
      "_id" : 0,
      "name" : "mongo-0-0:27018",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "optime" :

      { "t" : 1323358847000, "i" : 1 }

      ,
      "optimeDate" : ISODate("2011-12-08T15:40:47Z"),
      "self" : true
      },
      {
      "_id" : 1,
      "name" : "mongo-0-1:27018",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 513,
      "optime" :

      { "t" : 1323358847000, "i" : 1 }

      ,
      "optimeDate" : ISODate("2011-12-08T15:40:47Z"),
      "lastHeartbeat" : ISODate("2011-12-08T15:46:26Z"),
      "pingMs" : 2
      },
      {
      "_id" : 2,
      "name" : "mongo-0-0:27020",
      "health" : 1,
      "state" : 7,
      "stateStr" : "ARBITER",
      "uptime" : 327,
      "optime" :

      { "t" : 0, "i" : 0 }

      ,
      "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
      "lastHeartbeat" : ISODate("2011-12-08T15:46:25Z"),
      "pingMs" : 0
      }
      ],
      "ok" : 1
      }

      Then we go on router and flush router's config to update cached info:

      mongos> db.runCommand("flushRouterConfig")

      { "flushed" : true, "ok" : 1 }

      And the new node 'mongo-0-1' is not in shard-0:

      mongos> db.printShardingStatus()
      — Sharding Status —
      sharding version:

      { "_id" : 1, "version" : 3 }

      shards:

      { "_id" : "shard-0", "host" : "rs-0/mongo-0-0:27018" } { "_id" : "shard-1", "host" : "rs-1/mongo-1-0:27018" }

      databases:

      { "_id" : "admin", "partitioned" : false, "primary" : "config" }

      Then we check config-server, just to make sure that shard knows nothing about any new changes in replica set:

      db.shards.find()

      { "_id" : "shard-0", "host" : "rs-0/mongo-0-0:27018" } { "_id" : "shard-1", "host" : "rs-1/mongo-1-0:27018" }

      The only suspicious part in router's log was:

      Thu Dec 8 15:37:47 [ReplicaSetMonitorWatcher] updated set (rs-0) to: rs-0/mongo-0-0:27018,mongo-0-1:27018
      Thu Dec 8 15:37:47 [ReplicaSetMonitorWatcher] warning: node: mongo-0-1:27018 isn't a part of set: rs-0 ismaster:

      { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, ok: 1.0 }

      For now we decided to update shards collection manually on each change in replica set, but it would be great to delegate this job to mongo just as official documentation states.

            Assignee:
            greg_10gen Greg Studer
            Reporter:
            korsakov Dmytro Korsakov
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: