Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30705

Concurrent updates and FCV change can cause dbhash mismatch between primary and secondary

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.6.0-rc0
    • Affects Version/s: None
    • Component/s: Querying, Write Ops
    • None
    • Fully Compatible
    • ALL
    • Query 2017-08-21, Query 2017-09-11, Query 2017-10-02

      3.5.x versions of the server have two implementations of the update subsystem: the "old" (3.4 and earlier) system, and the new system in src/mongo/db/update which is both more performant and supports more expressive array updates. The old and new systems have different behavior with respect to field ordering. In order to ensure that the field ordering is consistent across all nodes in the replica set, the primary and secondaries must use the same version of the update subsystem. The is achieved via the feature compatibility version mechanism. Users must set the feature compatibility version (FCV) to "3.6" in order to enable the new update system.

      The FCV check, however, does not guarantee that a given update uses the same version of the update code on every node. Consider the following sequence of events:

      1. A two node replica set is started. Both nodes are version 3.6 but have FCV "3.4".
      2. The client concurrently issues an update and setFeatureCompatibilityVersion("3.6"). These operations take compatible locks, and therefore execute concurrently on the server.
      3. The setFCV command writes its update to admin.system.version to the oplog at optime t.
      4. After this oplog entry is written but before the in-memory FCV state changes, the update is logged with some optime greater than t. This uses the old update system, since the FCV in-memory state has not yet been changed.
      5. The two oplog entries are applied on the secondary. Since the admin.system.version write has an earlier optime (and must be applied in its own batch), the update uses the new update system.

      I was able to reproduce a dbhash mismatch against a two-node 3.5.x replica set by running two scripts concurrently from two shells connected to the primary node. The first script repeatedly issues an update with two $set's, that will result in different field ordering depending on which version of the update implementation is used:

      (function() {
          "use strict";
      
          db.c.drop();
          for (var i = 0; i < 1000; i++) {
              assert.writeOK(db.c.insert({_id: i}));
              assert.writeOK(db.c.update({_id: i}, {$set: {b: 1, a: 1}}));
          }
      }());
      

      The second script repeatedly sets the FCV from "3.4" to "3.6" and back again:

      (function() {
          "use strict";
      
          while (true) {
              assert.commandWorked(db.adminCommand({setFeatureCompatibilityVersion: "3.4"}))
              assert.commandWorked(db.adminCommand({setFeatureCompatibilityVersion: "3.6"}))
          }
      }());
      

      After the first script completes, running the dbHash command against the test database on each node should show different hashes for test.c.

            Assignee:
            justin.seyster@mongodb.com Justin Seyster
            Reporter:
            david.storch@mongodb.com David Storch
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: