Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-61803

Multiple Primary node in Mongodb replicaset

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.5
    • Component/s: None
    • Environment:
      Ubuntu 18.04.4 LTS

      Problem Statement/Rationale

      Sometimes our Mongodb replicaset has multiple primary nodes! in this case mongo-b node is our real primary and mongo-c node is not responding and just have primary role in fake.

      Number of nodes in replicaset: 4 (3 normal + 1 delay node without voting). network partition and access checked and there is no partion. we have had this problem several times since updating the cluster from version 4.2.5 to version 4.4.5. each time we fix the problem by cleaning datadir on fake node and restarting it.

      Steps to Reproduce

      it happens time to time. it doesn't have simple approach to Reproduce

      Expected Results

      cluster should have only one primary or produce better log about issue.

      Actual Results

      //                     {                                                                                                                [1881/1930]
                          "_id" : 5,                                       
                          "name" : "mongo-c:27017",                     
                          "health" : 1,
                          "state" : 1,
                          "stateStr" : "PRIMARY",
                          "uptime" : 3435,
                          "optime" : {
                                  "ts" : Timestamp(1637525581, 2),
                                  "t" : NumberLong(87)
                          },
                          "optimeDurable" : {
                                  "ts" : Timestamp(1637525443, 1),
                                  "t" : NumberLong(87)
                          },
                          "optimeDate" : ISODate("2021-11-21T20:13:01Z"),
                          "optimeDurableDate" : ISODate("2021-11-21T20:10:43Z"),
                          "lastHeartbeat" : ISODate("2021-11-24T07:08:32.108Z"),
                          "lastHeartbeatRecv" : ISODate("2021-11-24T07:08:34.867Z"),
                          "pingMs" : NumberLong(2355),
                          "lastHeartbeatMessage" : "",
                          "syncSourceHost" : "",
                          "syncSourceId" : -1,
                          "infoMessage" : "",
                          "electionTime" : Timestamp(1637524307, 1),
                          "electionDate" : ISODate("2021-11-21T19:51:47Z"),
                          "configVersion" : 23,
                          "configTerm" : 87
                  },
                  {
                          "_id" : 6,
                          "name" : "mongo-b:27017",
                          "health" : 1,
                          "state" : 1,
                          "stateStr" : "PRIMARY",
                          "uptime" : 7995003,
                          "optime" : {
                                  "ts" : Timestamp(1637737714, 24),
                                  "t" : NumberLong(88)
                          },
                          "optimeDate" : ISODate("2021-11-24T07:08:34Z"),
                          "syncSourceHost" : "",
                          "syncSourceId" : -1,
                          "infoMessage" : "",
                          "electionTime" : Timestamp(1637525581, 1),
                          "electionDate" : ISODate("2021-11-21T20:13:01Z"),
                          "configVersion" : 23,
                          "configTerm" : 88,
                          "self" : true,
                          "lastHeartbeatMessage" : ""
                  }
      

      Additional Notes

      replica conf:

                  {                                                                                                                   [25/599]
                          "_id" : 0,                                                                                                          
                          "host" : "mongo-a:27017",
                          "arbiterOnly" : false,
                          "buildIndexes" : true,
                          "hidden" : false,
                          "priority" : 1,
                          "tags" : {
      
                          },
                          "slaveDelay" : NumberLong(0),
                          "votes" : 1
                  },
                  {
                          "_id" : 3,
                          "host" : "mongo-h:27017",
                          "arbiterOnly" : false,
                          "buildIndexes" : true,
                          "hidden" : true,
                          "priority" : 0,
                          "tags" : {
      
                          },
                          "slaveDelay" : NumberLong(900),
                          "votes" : 0
                  },
                  {
                          "_id" : 5,
                          "host" : "mongo-c:27017",
                          "arbiterOnly" : false,
                          "buildIndexes" : true,
                          "hidden" : false,
                          "priority" : 1,
                          "tags" : {
      
                          },
                          "slaveDelay" : NumberLong(0),
                          "votes" : 1
                  },
                  {
                          "_id" : 6,
                          "host" : "mongo-b:27017",
                          "arbiterOnly" : false,
                          "buildIndexes" : true,
                          "hidden" : false,
                          "priority" : 1,
                          "tags" : {
      
                          },
                          "slaveDelay" : NumberLong(0),
                          "votes" : 1
                  }

            Assignee:
            edwin.zhou@mongodb.com Edwin Zhou
            Reporter:
            razzaghisaa@gmail.com sadegh raz
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: