-
Type: Improvement
-
Resolution: Unresolved
-
Priority: Unknown
-
None
-
Component/s: Client Side Encryption
-
None
-
Needed
Summary
Support auto-encryption with document sequences in libmongocrypt and drivers.
SERVER-40364 suggests mongocryptd already supports OP_MSG document sequence.
Consider requesting the the crypt_shared library support document sequences.
Motivation
I expect supporting a document sequence may benefit performance of auto encryption:
- May avoid copies of write payloads.
- May permit increasing the bulk write batch size for auto-encryption and reduce round trips.
Background
In the MongoDB wire protocol, a document sequence may be included as an OP_MSG payloadType=1 section as an optimization (documents are sent back-to-back, rather than nested in an array).
At present, the libmongocrypt and the crypt_shared components of In-Use Encryption (CSFLE/QE) do not support document sequences.
If a driver constructs a document sequence for a write, the document sequence must be converted to a BSON array prior to sending to libmongocrypt. Example:
Driver constructs a document sequence for an insert to be sent in an OP_MSG payloadType=1:
{ "_id": 1, "encrypted": "foo" } { "_id": 2, "encrypted": "bar" }
Automatic encryption does not support document sequence. The document sequence must be converted to a BSON array prior to passing to libmongocrypt:
{ "insert": "coll", "documents": [ { "_id": 1, "encrypted": "foo" }, { "_id": 2, "encrypted": "bar" } ] }
For example, the C driver has an internal function to conversion.
The maximum size of a payloadType=0 payload is ~16MiB. As a consequence, bulk writes for automatic encryption are split at a reduced batch size of 2MiB. The maximum size of a payloadType=1 payload is ~48MiB.
Who is the affected end user?
Users of QE/CSFLE.
How does this affect the end user?
This change may improve write performance.
How likely is it that this problem or use case will occur?
Likely. Suggest resolving DRIVERS-2669 first (Add performance benchmarks for FLE/QE) to help measure a proposed improvement.
Is this issue urgent?
No.
Is this ticket required by a downstream team?
No.
Is this ticket only for tests?
No.
Acceptance Criteria
- Update libmongocrypt to support document sequence for automatic encryption.
- Update drivers to support passing document sequences to libmongocrypt for automatic encryption.