Interface for time-series bucket access through user namespace

XMLWordPrintableJSON

    • Needed
    • Hide

      Before this project, it was possible to access timeseries data in the compressed format (buckets) by directly performing read/write on the bucket collection (<db>.system.buckets.<coll>).

      This is not possible anymore. In fact, the bucket collection does not exist anymore. Instead, to access timeseries data in the compressed format (buckets) the CRUD operations must target the main timeseries user namespace and pass the "rawData: true" parameter.

      E.g.:
      db.runCommand(

      {find: 'coll', rawData: true}

      )

      Show
      Before this project, it was possible to access timeseries data in the compressed format (buckets) by directly performing read/write on the bucket collection (<db>.system.buckets.<coll>). This is not possible anymore. In fact, the bucket collection does not exist anymore. Instead, to access timeseries data in the compressed format (buckets) the CRUD operations must target the main timeseries user namespace and pass the "rawData: true" parameter. E.g.: db.runCommand( {find: 'coll', rawData: true} )
    • Hide

      Summary of necessary driver changes

      •  Add support for the new rawData command option for the following CRUD operations:
        • aggregate
        • bulkWrite (collection and client)
        • count
        • countDocuments
        • deleteMany
        • deleteOne
        • distinct
        • estimatedDocumentCount
        • find
        • findOne
        • findAndModify
        • insertMany
        • insertOne
        • replaceOne
        • updateMany
        • updateOne

      Note: the list of CRUD operations that need to support this new option was derived from the list of commands that support the option:

      • aggregate
      • bulkWrite
      • count
      • delete
      • distinct
      • find
      • findAndModify
      • insert
      • update

      Commits for syncing spec/prose tests
      (and/or refer to an existing language POC if needed)

      •  

      Context for other referenced/linked tickets

      •  
      Show
      Summary of necessary driver changes  Add support for the new rawData command option for the following CRUD operations: aggregate bulkWrite (collection and client) count countDocuments deleteMany deleteOne distinct estimatedDocumentCount find findOne findAndModify insertMany insertOne replaceOne updateMany updateOne Note: the list of CRUD operations that need to support this new option was derived from the list of commands that support the option: aggregate bulkWrite count delete distinct find findAndModify insert update Commits for syncing spec/prose tests (and/or refer to an existing language POC if needed)   Context for other referenced/linked tickets  
    • $i18n.getText("admin.common.words.hide")
      Key Status/Resolution FixVersion
      CDRIVER-5955 Won't Fix
      CXX-3258 Won't Fix
      CSHARP-5546 Won't Fix
      GODRIVER-3522 Investigating
      JAVA-5830 Won't Fix
      NODE-6883 Blocked
      MOTOR-1447 Won't Fix
      PYTHON-5247 Won't Fix
      PHPLIB-1655 Won't Fix
      RUBY-3639 Won't Fix
      RUST-2189 Won't Fix
      $i18n.getText("admin.common.words.show")
      #scriptField, #scriptField *{ border: 1px solid black; } #scriptField{ border-collapse: collapse; } #scriptField td { text-align: center; /* Center-align text in table cells */ } #scriptField td.key { text-align: left; /* Left-align text in the Key column */ } #scriptField a { text-decoration: none; /* Remove underlines from links */ border: none; /* Remove border from links */ } /* Add green background color to cells with FixVersion */ #scriptField td.hasFixVersion { background-color: #00FF00; /* Green color code */ } #scriptField td.willNotDo { background-color: #FF0000; /* Red color code */ } /* Center-align the first row headers */ #scriptField th { text-align: center; } Key Status/Resolution FixVersion CDRIVER-5955 Won't Fix CXX-3258 Won't Fix CSHARP-5546 Won't Fix GODRIVER-3522 Investigating JAVA-5830 Won't Fix NODE-6883 Blocked MOTOR-1447 Won't Fix PYTHON-5247 Won't Fix PHPLIB-1655 Won't Fix RUBY-3639 Won't Fix RUST-2189 Won't Fix

      Original Downstream Change Summary

      Before this project, it was possible to access timeseries data in the compressed format (buckets) by directly performing read/write on the bucket collection (<db>.system.buckets.<coll>).

      This is not possible anymore. In fact, the bucket collection does not exist anymore. Instead, to access timeseries data in the compressed format (buckets) the CRUD operations must target the main timeseries user namespace and pass the "rawData: true" parameter.

      E.g.:

      db.runCommand({find: 'coll', rawData: true})

      Description of Linked Ticket

      Epic Summary

      Summary

      Implement "raw access" mode for CRUD commands, which bypasses any logical translation steps. For standard collections, this will be a noop. For time-series collections, it allows operations to function on the bucket-level data while accessing the primary namespace (not system.buckets). This will allow tooling such as mongodump to access the correct data format without knowing the collection type.

      Motivation

      This small project will be a preparatory step for SPM-3830 (Eliminate Dual-Namespaces for Time Series Collections) to ease the transition to using a single namespace for time-series collections.

      Documentation

      Product Description
      Scope
      Technical Design
      Docs Update

      Discussion from Slack

      The requirements we took into accounts are:

      • The server must explicitly reject requests with rawData=true on commands that do not support this parameter. It is important that the server does not silently ignore this parameter.
      • If in the future we decide to add rawData support on a command that previously was refusing it, this will cause a change of behavior for end-users (applications). So we must ensure that the change in behavior only happens when the customer decide to upgrade their driver version and not when they upgrade the server.
      • A change in the list of commands that supports rawData must not require a change in the application code.
      • Even in the initial release in a driver targetting 9.0, the addition of rawData to a command document should be guarded by a wire version check. That will allow tools like mongodump/restore to always enable rawData at the client level, and the driver will just ignore it when connected to pre-9.0 servers.

      And the proposal we agreed on is the following:

      • The server will accept rawData parameter only on a subset of the commands and for all the others were rawData does not have any effect or is not applicable, the server will return InvalidOptions error.
      • The drivers will allow to specify the rawData parameter through some high/gloabal level object (e.g. Client or connection object).
      • The drivers will need to maintain a list of commands that accepts the rawData parameter and forward the parameter only to those.

            Assignee:
            Qingyang Hu
            Reporter:
            Tom Selander
            Andreas Braun Andreas Braun
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: