Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-56950

Avoid shardRegistry reload infinite loop when overlapping with setFCV

    • Fully Compatible
    • ALL
    • v5.0
    • Hide

      1.  ShardRegistry::_periodicReload causes a reload to occur. ShardRegistry::_getDataAsync advances the ReadThroughCache's timeInStore to some t1 with non-zero topologyTime. ReadThroughCache::acquireAsync creates an inProgressLookup with t1, and add a promise for it to inProgressLookup._outstanding.

      2. ShardRegistry::_lookup starts running, and meanwhile the test runs setFCV from 4.9 to 4.4.

      3. ShardRegistryData::createFromCatalogClient returns. useActualTopologyTime() is false so it returns the cached data's topology time (i.e. Timestamp(0,0) since this is the first reload) as result.t. 

      4. Inside ReadThroughCache::_doLookupWhileNotValid, inProgressLookup.getPromisesLessThanTime returns nothing because the first promise in _outstanding is the promise for t1 which has non-zero topologyTime (i.e. t1 > result.t) so the for loop breaks early here.

      5. The promisesToSet is empty so mustDoAnotherLoop is true. The _inProgressLookup for t1 remains in the cache, and another round of lookup starts, again no promises can be fulfilled because of 4.

      6. Future reloads join this infinitely looping inProgressLookup. (That's why in the hang analyzer output, there are multiple mongo::ShardRegistry::_periodicReload threads).

      Show
      1.  ShardRegistry::_periodicReload causes a reload to occur. ShardRegistry::_getDataAsync advances the ReadThroughCache's timeInStore to some t1 with non-zero topologyTime. ReadThroughCache::acquireAsync creates an inProgressLookup with t1, and add a promise for it to  inProgressLookup._outstanding . 2. ShardRegistry::_lookup starts running, and meanwhile the test runs setFCV from 4.9 to 4.4. 3. ShardRegistryData::createFromCatalogClient returns. useActualTopologyTime() is false so it returns the cached data's topology time (i.e. Timestamp(0,0) since this is the first reload) as result.t.  4. Inside ReadThroughCache::_doLookupWhileNotValid, inProgressLookup.getPromisesLessThanTime returns nothing because the first promise in _outstanding is the promise for t1 which has non-zero topologyTime (i.e.  t1 > result.t ) so the for loop breaks early here . 5. The promisesToSet is empty so mustDoAnotherLoop is true. The _inProgressLookup for t1 remains in the cache, and another round of lookup starts, again no promises can be fulfilled because of 4. 6. Future reloads join this infinitely looping inProgressLookup. (That's why in the hang analyzer output, there are multiple mongo::ShardRegistry::_periodicReload threads).
    • Sharding EMEA 2021-05-31
    • 170

      When setFCV(v4.4) overlaps with a ShardRegistry reload - right after the useActualTopologyTime check - the ShardRegistry can fall into an infinite loop of lookups because the topology time is not gossiped after the setFCV succeeds.

      Purpose of this ticket is to avoid this overlap to result in a livelock.

            Assignee:
            simon.gratzer@mongodb.com Simon Gratzer (Inactive)
            Reporter:
            pierlauro.sciarelli@mongodb.com Pierlauro Sciarelli
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: