Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-95461

Improve cross-shard updates in change stream.

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution

      Consider an update that changes a document’s shard key, such that the document moves from one shard to another. Currently change streams show such updates as 2 separate events: delete and insert. Those events usually appear in that order but can appear as insert-then-delete.

      TAR assumed, when designing mongosync’s embedded verifier, that the delete would always precede the insert, so we’re now having to implement a workaround. (cf. REP-4857)

      Even if the server guaranteed delete-then-insert, though, this would still be suboptimal because, to a change stream reader, the document in question will appear to be “temporarily missing” during the time period—however short—between the delete and insert. For real-time applications this could cause durability problems.

      In a Slack thread just now with ivan.fefer@mongodb.com and mickey.winters@mongodb.com, an alternate solution arose. If we can update the oplog format to indicate which insert & delete operations relate to shard-key updates, it may be possible instead to:

      1. Omit the oplog’s delete from the change stream, and
      2. Report the oplog’s insert as a replace event.

      This way, change stream consumers only see a single replace event. This solves both the ordering and “temporarily missing” problems.

      (NB: the oplog’s existing fromMigrate flag concerns chunk migrations, not shard-key updates, so it’s likely unhelpful here.)

      cc max.hirschhorn@mongodb.com, michael.mcclimon@mongodb.com, craven.huynh@mongodb.com

            Assignee:
            Unassigned Unassigned
            Reporter:
            felipe.gasper@mongodb.com Felipe Gasper
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: