Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-49891

Write and test aggregation pipeline for oplog fetcher for resharding

    • Fully Compatible
    • Sharding 2020-09-21

      The goal of this ticket is to create a function which allows the aggregation pipeline to easily be sent to a remote (donor) shard and for the aggregation pipeline to be unit-testable with DocumentSourceMock. It is preferable to use DocumentSourceXX::create() functions (or DocumentSourceXX::parseFromBSON() when the former isn't available or is too tedious) rather than building it up with string concatenation. DocumentSources can be conditionally added to the Pipeline::SourceContainer, for example, to reflect a stage being added only when resuming on a new cursor.

      std::unique_ptr<Pipeline, PipelineDeleter> createOplogFetchingPipelineForResharding(
          ReshardingDonorOplogId startAfter,  /* isNull() when not resuming */
          ShardId recipientShard
      );
      

      Some of these parameters are probably more appropriate to take by const-ref because their contents can only be copied into the Pipeline anyway.


      Testing should cover the following types of oplog entries:

      • Ordinary and retryable insert, update, or delete.
        • Includes formats for $set/$unset updates, replacement updates, and $v:2 delta updates (compressed pipeline updates).
      • The preImageOpTime and postImageOpTime documents from retryable findAndModify operations.
      • The final oplog entry marker from when the donor shard entered the "all new incoming writes as distributed transactions" state.
      • Drop collection.
      • All of the above for the matching (namespace, recipient), a different recipient, and a different namespace.

            Assignee:
            daniel.gottlieb@mongodb.com Daniel Gottlieb (Inactive)
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: