Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-2859

Support auto-encryption with document sequences

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • None
    • Component/s: Client Side Encryption
    • None
    • Needed

      Summary

      Support auto-encryption with document sequences in libmongocrypt and drivers.

      SERVER-40364 suggests mongocryptd already supports OP_MSG document sequence.

      Consider requesting the the crypt_shared library support document sequences.

      Motivation

      I expect supporting a document sequence may benefit performance of auto encryption:

      • May avoid copies of write payloads.
      • May permit increasing the bulk write batch size for auto-encryption and reduce round trips.

      Background

      In the MongoDB wire protocol, a document sequence may be included as an OP_MSG payloadType=1 section as an optimization (documents are sent back-to-back, rather than nested in an array).

      At present, the libmongocrypt and the crypt_shared components of In-Use Encryption (CSFLE/QE) do not support document sequences.

      If a driver constructs a document sequence for a write, the document sequence must be converted to a BSON array prior to sending to libmongocrypt. Example:

      Driver constructs a document sequence for an insert to be sent in an OP_MSG payloadType=1:

      { "_id": 1, "encrypted": "foo" }
      { "_id": 2, "encrypted": "bar" }
      

      Automatic encryption does not support document sequence. The document sequence must be converted to a BSON array prior to passing to libmongocrypt:

      {
          "insert": "coll",
          "documents": [
              { "_id": 1, "encrypted": "foo" },
              { "_id": 2, "encrypted": "bar" }
          ]
      }
      

      For example, the C driver has an internal function to conversion.

      The maximum size of a payloadType=0 payload is ~16MiB. As a consequence, bulk writes for automatic encryption are split at a reduced batch size of 2MiB. The maximum size of a payloadType=1 payload is ~48MiB.

      Who is the affected end user?

      Users of QE/CSFLE.

      How does this affect the end user?

      This change may improve write performance.

      How likely is it that this problem or use case will occur?

      Likely. Suggest resolving DRIVERS-2669 first (Add performance benchmarks for FLE/QE) to help measure a proposed improvement.

      Is this issue urgent?

      No.

      Is this ticket required by a downstream team?

      No.

      Is this ticket only for tests?

      No.

      Acceptance Criteria

      • Update libmongocrypt to support document sequence for automatic encryption.
      • Update drivers to support passing document sequences to libmongocrypt for automatic encryption.

            Assignee:
            Unassigned Unassigned
            Reporter:
            kevin.albertson@mongodb.com Kevin Albertson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: