-
Type: Improvement
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
Consider an update that changes a document’s shard key, such that the document moves from one shard to another. Currently change streams show such updates as 2 separate events: delete and insert. Those events usually appear in that order but can appear as insert-then-delete.
TAR assumed, when designing mongosync’s embedded verifier, that the delete would always precede the insert, so we’re now having to implement a workaround. (cf. REP-4857)
Even if the server guaranteed delete-then-insert, though, this would still be suboptimal because, to a change stream reader, the document in question will appear to be “temporarily missing” during the time period—however short—between the delete and insert. For real-time applications this could cause durability problems.
In a Slack thread just now with ivan.fefer@mongodb.com and mickey.winters@mongodb.com, an alternate solution arose. If we can update the oplog format to indicate which insert & delete operations relate to shard-key updates, it may be possible instead to:
- Omit the oplog’s delete from the change stream, and
- Report the oplog’s insert as a replace event.
This way, change stream consumers only see a single replace event. This solves both the ordering and “temporarily missing” problems.
(NB: the oplog’s existing fromMigrate flag concerns chunk migrations, not shard-key updates, so it’s likely unhelpful here.)
cc max.hirschhorn@mongodb.com, michael.mcclimon@mongodb.com, craven.huynh@mongodb.com
- is related to
-
SERVER-70156 Allow updateLookup to succeed for updates which modify the shard key
- Backlog