Uploaded image for project: 'Node.js Driver'
  1. Node.js Driver
  2. NODE-923

After adding new secondaries and flipping primary, indefinitely reconnects

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.2.24
    • Affects Version/s: 2.2.21
    • Component/s: MongoDB 3.2
    • Environment:
      CentOS 7. Node.JS 6.9.4. Mongo 3.2

      We have the following logging in place:

      topology.on('left', function(type, server) {
  log.warn('lost connection to mongo server', {type: type, name: server.name || server.me});
}.bind(this));
topology.on('joined', function(type, server) {
  log.info('new connection to mongo server', {type: type, name: server.name || server.me});
}.bind(this));

      This afternoon, we added 3 new secondaries, stan, turk, and regan. The primary at the time was barack. Then later, we swapped barack and regan so now regan was the primary. Instantly we started seeing logs from all of our Node.js applications about disconncting and reconnecting. Below is just from one app on one server:

      Jan 31 21:17:24 WARN -- lost connection to mongo server -- type="primary" name="barack.nodes.admiral:27017"
Jan 31 21:17:25 WARN -- lost connection to mongo server -- type="secondary" name="regan.nodes.admiral:27017"
Jan 31 21:17:26 INFO -- new connection to mongo server -- type="secondary" name="regan.nodes.admiral:27017"
Jan 31 21:17:26 INFO -- new connection to mongo server -- type="secondary" name="barack.nodes.admiral:27017"
Jan 31 21:17:41 WARN -- lost connection to mongo server -- type="secondary" name="stan.nodes.admiral:27017"
Jan 31 21:17:41 INFO -- new connection to mongo server -- type="secondary" name="stan.nodes.admiral:27017"
Jan 31 21:17:51 WARN -- lost connection to mongo server -- type="secondary" name="turk.nodes.admiral:27017"
Jan 31 21:17:51 INFO -- new connection to mongo server -- type="secondary" name="turk.nodes.admiral:27017"
Jan 31 21:17:56 WARN -- lost connection to mongo server -- type="secondary" name="regan.nodes.admiral:27017"
Jan 31 21:17:57 INFO -- new connection to mongo server -- type="primary" name="regan.nodes.admiral:27017"
Jan 31 21:18:11 WARN -- lost connection to mongo server -- type="secondary" name="stan.nodes.admiral:27017"
Jan 31 21:18:12 INFO -- new connection to mongo server -- type="secondary" name="stan.nodes.admiral:27017"
Jan 31 21:18:21 WARN -- lost connection to mongo server -- type="secondary" name="turk.nodes.admiral:27017"
Jan 31 21:18:23 INFO -- new connection to mongo server -- type="secondary" name="turk.nodes.admiral:27017"
Jan 31 21:18:27 WARN -- lost connection to mongo server -- type="primary" name="regan.nodes.admiral:27017"
Jan 31 21:18:28 INFO -- new connection to mongo server -- type="primary" name="regan.nodes.admiral:27017"
Jan 31 21:18:42 WARN -- lost connection to mongo server -- type="secondary" name="stan.nodes.admiral:27017"
Jan 31 21:18:44 INFO -- new connection to mongo server -- type="secondary" name="stan.nodes.admiral:27017"
Jan 31 21:18:53 WARN -- lost connection to mongo server -- type="secondary" name="turk.nodes.admiral:27017"
Jan 31 21:18:54 INFO -- new connection to mongo server -- type="secondary" name="turk.nodes.admiral:27017"
Jan 31 21:18:58 WARN -- lost connection to mongo server -- type="primary" name="regan.nodes.admiral:27017"
Jan 31 21:18:59 INFO -- new connection to mongo server -- type="primary" name="regan.nodes.admiral:27017"

      The applications would continue logging about this multiple times a minute indefinitely until restarted, which after restart they worked fine and don't log any disconnects at all. The applications were started before the new secondaries were in the cluster, if that helps.
      Also, barack was not shutdown until 30 minutes after stepping down.

      Is there anything else I can provide to help this issue?

            Assignee:
            Unassigned Unassigned
            Reporter:
            fastest963 James Hartig
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: