Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-60774

Resharding may apply through reshardFinalOp without transitioning to strict consistency, stalling write operations on collection being resharded until critical section times out

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 5.2.0, 5.0.4, 5.1.0-rc2
    • Affects Version/s: 5.1.0-rc0
    • Component/s: Sharding
    • None
    • Fully Compatible
    • ALL
    • v5.1, v5.0
    • Sharding 2021-11-01
    • 1

      Typically the ReshardingOplogFetcher stops fetching new donor oplog entries after it has retrieved the reshardFinalOp entry. However, upon resuming from a primary failover, the ReshardingOplogFetcher won't realize it has fetched the reshardFinalOp entry already and it'll continue to retrieve empty batches. (The donor shard won't write any new oplog entries destined for the recipient shard after the reshardFinalOp.) This leads the ReshardingOplogFetcher to insert no-op reshardProgressMark entries after the reshardFinalOp intos the recipient shard's local oplog buffer collection.

      The ReshardingDonorOplogIterator assumes the reshardFinalOp will be the last entry in a batch. The presence of these extra no-op reshardProgressMark entries after the reshardFinalOp will prevent it and the ReshardingOplogApplier from realizing the reshardFinalOp has been applied through. The recipient shard therefore never reaches the "strict-consistency" state. This leads the overall resharding operation to fail with a ReshardingCriticalSectionTimeout error response after writes been blocked for the collection being resharded for reshardingCriticalSectionTimeoutMillis (5 seconds by default).

      if (!batch.empty()) {
          const auto& lastEntryInBatch = batch.back();
          _resumeToken = getId(lastEntryInBatch);
      
          if (isFinalOplog(lastEntryInBatch)) {
              _hasSeenFinalOplogEntry = true;
              // Skip returning the final oplog entry because it is known to be a no-op.
              batch.pop_back();
      

            Assignee:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: