-
Type: New Feature
-
Resolution: Unresolved
-
Priority: Unknown
-
None
-
Component/s: Change Streams
-
None
-
Needed
Summary
Add option to resume change stream after receiving an "invalidate" event.
Motivation
After a change stream receives an "invalidate" event, the server closes the change stream.
Callers wanting to continue receiving events are required to create a new change stream with the `startAfter` option set to the most recent `resumeToken`.
An opt-in option (e.g. `ChangeStreamOptions.resumeAfterInvalidate`?) to resume using `startAfter` may provide a convenience.
Who is the affected end user?
Change stream users who want to continue listening for events after "invalidate".
How does this affect the end user?
To achieve this behavior, it requires a workaround: Track the resume token. If an "invalidate" event is received, recreate the change stream with the `startAfter` option.
How likely is it that this problem or use case will occur?
Not sure.
If the problem does occur, what are the consequences and how severe are they?
Requires a workaround.
Is this issue urgent?
No. There is a workaround.
Is this ticket required by a downstream team?
Requested by internal team described in this comment.
Is this ticket only for tests?
No.
Acceptance Criteria
Enable change streams to continue listening after receiving an "invalidate" event.
Original ticket description:
To have a change stream survive after an “invalidate”, I expect it is needed to recreate the change stream with the “startAfter” option.Modifying the example, this behavior reproduces:
- Insert a document to db.coll to create the collection.
- Watch on db.coll.
- Drop db.coll.
- Iterate. Get a “drop” event.
- Iterate. Get an “invalidate” event.
- Iterate. Get an error: “Attempting to resume a change stream using ‘resumeAfter’ is not allowed from an invalidate notification”.The change stream specification describes the expected protocol. Drivers do not inspect events or handle “invalidate” specially.To have a change stream survive after an “invalidate”, I expect it is needed to recreate the change stream with the “startAfter” option.
Kevin Albertson 2 hours agoI think there is a related bug in the C driver. I do not expect the resume after “invalidate”. Comparing with the Go driver, I expect iterating after “invalidate” should result in no event returned and no error. I plan to file a ticket.
matthew.normyle 37 minutes agoThank you so much for investigating.
Yep, for what it’s worth, this problem goes away if I change this line to startAfter instead of resumeAfter
mongoc-change-stream.c
&change_stream_doc, "resumeAfter", &stream->resume_token);
mongodb/mongo-c-driver | Added by GitHub
Kevin Albertson 25 minutes agoI expect the Resume Process using resumeAfter rather than startAfter is intentional. After “invalidate”, later events may not be related to previous events. E.g. the collection may have been dropped, and recreated with a different schema. It may no longer be the “same collection”.
matthew.normyle 24 minutes agoYep, totally makes sense. Perhaps the fix here is to expose an option for this, so the customer can decide what happens. In our case we’d like to keep reading after a drop and invalidate event (startAfter semantics).
matthew.normyle 24 minutes ago(Also I think startAfter only works for MongoDB >= 4.X)
Kevin Albertson 22 minutes agoThat seems reasonable. If it is wanted, I suggest filing a DRIVERS ticket with the feature request. That may result in a specification change to support that use case.
- is related to
-
CDRIVER-4738 Do not resume after change stream cursor is closed
- Backlog