Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34672

Unable to add shard on 3.7.5 sharded cluster with mmapv1 shard

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • None
    • ALL
    • Hide
      1. Start up a 3-node shard on mmapv1
      2. Initiate repl set: 
        > rs.initiate({_id: "a", "members" : [ {"_id" : 0, host : "louisamac:9001"}, {_id: 1, host: "louisamac:9002"}, {_id: 2, host: "louisamac:9003"} ] })
{
	"ok" : 1,
	"operationTime" : Timestamp(1524673861, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524673861, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}
      3. Start up a 3-node csrs
      4. Initiate csrs 
        > rs.initiate({configsvr: true, protocolVersion:1, _id: "csrs", "members" : [ {"_id" : 0, host : "louisamac:9007"}, {_id: 1, host: "louisamac:9008"}, {_id: 2, host: "louisamac:9009"} ] })
{
	"ok" : 1,
	"operationTime" : Timestamp(1524674021, 1),
	"$gleStats" : {
		"lastOpTime" : Timestamp(1524674021, 1),
		"electionId" : ObjectId("000000000000000000000000")
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524674021, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"lastCommittedOpTime" : Timestamp(0, 0)
}
      5. Start up mongos
      6. Call {{db.runCommand( {"setFeatureCompatibilityVersion" : "4.0"}

        )}} on mongos

      7. Attempt to add a shard 
        mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"})
{
	"ok" : 0,
	"errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out",
	"code" : 96,
	"codeName" : "OperationFailed",
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524674234, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"operationTime" : Timestamp(1524674234, 1)
}
      Show
      Start up a 3-node shard on mmapv1 Initiate repl set: > rs.initiate({_id: "a" , "members" : [ { "_id" : 0, host : "louisamac:9001" }, {_id: 1, host: "louisamac:9002" }, {_id: 2, host: "louisamac:9003" } ] }) { "ok" : 1, "operationTime" : Timestamp(1524673861, 1), "$clusterTime" : { "clusterTime" : Timestamp(1524673861, 1), "signature" : { "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA=" ), "keyId" : NumberLong(0) } } } Start up a 3-node csrs Initiate csrs > rs.initiate({configsvr: true , protocolVersion:1, _id: "csrs" , "members" : [ { "_id" : 0, host : "louisamac:9007" }, {_id: 1, host: "louisamac:9008" }, {_id: 2, host: "louisamac:9009" } ] }) { "ok" : 1, "operationTime" : Timestamp(1524674021, 1), "$gleStats" : { "lastOpTime" : Timestamp(1524674021, 1), "electionId" : ObjectId( "000000000000000000000000" ) }, "$clusterTime" : { "clusterTime" : Timestamp(1524674021, 1), "signature" : { "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA=" ), "keyId" : NumberLong(0) } }, "lastCommittedOpTime" : Timestamp(0, 0) } Start up mongos Call {{db.runCommand( {"setFeatureCompatibilityVersion" : "4.0"} )}} on mongos Attempt to add a shard mongos> db.runCommand({ "addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003" , name: "shard0" }) { "ok" : 0, "errmsg" : "failed to run command { setFeatureCompatibilityVersion: \" 4.0\ " } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out" , "code" : 96, "codeName" : "OperationFailed" , "$clusterTime" : { "clusterTime" : Timestamp(1524674234, 1), "signature" : { "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA=" ), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1524674234, 1) }
    • 0

      If you bring up a sharded cluster on 3.7.5 with a mmapv1 multi-node shard, you get a timeout error from trying to add the shard on the mongos: 

      mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"})
{
	"ok" : 0,
	"errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out",
	"code" : 96,
	"codeName" : "OperationFailed",
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524674234, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"operationTime" : Timestamp(1524674234, 1)
}

      Note: I tried this on WT and it worked fine. Assigning to storage for that reason, but may need to be reassigned.
      Second note: Also did not have this problem with a 1-node shard.

      Attached config files and log files.

        1. cfg1.json
          0.4 kB
        2. cfg2.json
          0.4 kB
        3. cfg3.json
          0.4 kB
        4. mongos.json
          0.2 kB
        5. shard1.json
          0.4 kB
        6. shard2.json
          0.4 kB
        7. shard3.json
          0.4 kB
        8. mongos.log
          7 kB
        9. csrs_primary.log
          41 kB
        10. csrs_secondary.log
          29 kB
        11. shard_primary.log
          37 kB
        12. shard_secondary.log
          37 kB

            Assignee:
            esha.maharishi@mongodb.com Esha Maharishi (Inactive)
            Reporter:
            louisa.berger@mongodb.com Louisa Berger
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: