Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54167

configOpTime can become ahead of VectorClock::clusterTime

    • Fully Compatible
    • ALL
    • Hide

      Run the sharding suite with the changes in this patch that adds an invariant:

       

      invariant(configTime.getTimestamp() <= currentTime.clusterTime().asTimestamp());
      

      in Grid.cpp and check that it gets tripped.

       

       

      Show
      Run the sharding suite with the changes in  this patch  that adds an invariant:   invariant(configTime.getTimestamp() <= currentTime.clusterTime().asTimestamp()); in Grid.cpp and check that it gets tripped.    
    • Sharding 2021-02-08
    • 45

      The configOpTime (Grid::configOpTime()) can get ahead of the Vector's Clock clusterTime following the this sequence of events:

      1. Node A is sending a request to node B.
      2. In A, when preparing the request metadata, it will first get the vector clock components and add them to the request.
      3. Concurrently, another thread in A bumps the clocks (vector clock & configOpTime). Now configOpTime is greater than the clusterTime that we read at point 2
      4. A adds the configOpTime to the request metadata (which because of point 3, is greater than the clusterTime it wrote at 2
      5. The request gets sent to node B, which updates it's clock with the received times. B is left with a configOpTime greter than VectorClock::clusterTime
      6. When something later calls Grid::configOpTime, the invariant is tripped.

      The order in which the metadata egress hooks are called is set here for the mongos and here for mongod. Changing the hooks order so that the configOpTime hook runs before the vectorClock hook would not solve the issue, because of the fact that the hooks are run on the same order when reading reply metadata. If we did so, then the configOpTime would get advanced before VectorClock, leading to the same situation.

            Assignee:
            jordi.serra-torrens@mongodb.com Jordi Serra Torrens
            Reporter:
            jordi.serra-torrens@mongodb.com Jordi Serra Torrens
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: