Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-68341

Implement enable/disable command for mongoQ in serverless

    • Query Execution

      PM-2334 implemented enable/disable change streams command for the replica sets. The corresponding ticket was: SERVER-66631

      This work for the sharded cluster was punted because there are subtle differences in how mongoS and mongoQ behave. For example, in mongoQ, the above command should dispatch the request to only a subset of shards where the corresponding tenant resides as opposed to mongoS where the request gets sent to all the shards. Since, there can be up to 10k shards (nodes) in the serverless, fanning out requests blindly to all the nodes is not a scalable solution. This ticket is about extending the work done in the SERVER-66631 to run in the mongoQ.

       

      Since we will be building this enable/disable command using the POS service, we don't have atomicity when creating change collections on shards, ie. a change collection at shard1 might be made a timestamp t1 and on shard2 at timestamp t2. Whilst in general, this is not a problem for the change stream, it could be a problem for the following rare occurring scenario:

      Consider this:

      1. shard1 change collection creating ts is t1
      2. shard2 change collection creating rs is t2
      3.  t1 < t2
      4. other than the change collection seed-entry (own create-collection oplog entry), no other entries were written to the change collection, ie. each of the change collections has just one entry, ie the seed-entry.
      5. the client immediately opened the change stream cursor and the resume token corresponds to t1.
      6. a get-next on shard2 will result in a ChangeStreamHistoryLost error, as the time t2 > provided time t1.
      7. this case is similar to the addShard case with the change collection.

       

      Above is not a problem for the oplog because the oplog gets created at a very early stage of the node-bring up, as such the change stream resume token is always greater than the first entry of all the oplog in all the nodes. 

      To mitigate above issue, we could potentially enable a change stream in the mongoQ in 2 stages -

      1. the first stage will be to create a change collection in required shards.
      2. And the second stage will be to add a dummy oplog entry to the tenant's change collection in required shards.

      With this, we will ensure that the resume token that the client will operate upon will be greater than the initial timestamp of all the change collections.

      While this is just one solution, we will explore more options when we work on this ticket.

       

      This work will be under the serverless initiative - Change streams with mongoq

            Assignee:
            backlog-query-execution [DO NOT USE] Backlog - Query Execution
            Reporter:
            rishab.joshi@mongodb.com Rishab Joshi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: