Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-89286

Implicit FLE state collection creation fails when data placed outside the db primary shard

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 6.0.0, 7.0.0, 8.0.0-rc0
    • Component/s: None
    • Catalog and Routing
    • ALL
    • CAR Team 2024-09-16, CAR Team 2024-09-30, CAR Team 2024-10-14, CAR Team 2024-10-28, CAR Team 2024-11-11, CAR Team 2024-11-25, CAR Team 2024-12-09
    • 3

      Collection creation in transactions involving more than one shard is disallowed because there is no conflict detection machinery at prepare time for distributed transactions (SERVER-46105 should address that).

      Write operations on encrypted fields are transactionally updating both main collection (already existing) and state collections (lazily created). Because of that, the first insert on a collection with FLE options is triggering the creation of the necessary state collections within a transaction.

      Example of faulty flow:

      • Create FLE collection
      • Move it out the primary shard (moveChunk for sharded collection / moveCollection for unshareded collection)
      • Insert data
        • Write on the main collection will be routed to a shard different than the db primary
        • Write on state collection(s) will target the primary shard and implicitly create the collection
        • Transaction fails because trying to create collection in a distributed transaction

      Possible solutions:

      • Catch the error in fle write path and create FLE collections before retrying the transaction
      • Always create FLE state collections as part of the main collection creation

      Reproducible:

      const st = new ShardingTest({shards: 2, mongos: 1, rs: {nodes: 1}});
const mongos = st.s0;
const shard0 = st.shard0.shardName;
const shard1 = st.shard1.shardName;

const kDbName = 'test';
const kCollName = 'coll';
const kFullNs = kDbName + '.' + kCollName;

const kms = {
    key: BinData(
        0,
        "/tu9jUCBqZdwCelwE/EAm/4WqdxrSMi04B8e9uAV+m30rI1J2nhKZZtQjdvsSCwuI4erR6IEcEK+5eGUAODv43NDNIR9QheT2edWFewUfHKsl9cnzTc86meIzOmYl6dr")
};

const csfleOpts = {
    kmsProviders: {
        local: kms,
    },
    keyVaultNamespace: kDbName + ".keystore",
    schemaMap: {},
};

var shell = Mongo(mongos.host, csfleOpts);
var kv = shell.getKeyVault();

const schema = {
    "fields": [
        {
            "path": "foo",
            "keyId": kv.createKey("local", "ignored"),
            "bsonType": "string",
            "queries": {"queryType": "equality"}
        },
    ]
};

// Create FLE collection, shard it and move unique range out of the db primary shard
const db = shell.getDB(kDbName);
assert.commandWorked(db.createCollection(kCollName, {encryptedFields: schema}));
assert.commandWorked(mongos.adminCommand({shardCollection: kFullNs, key: {_id: 1}}));
const toShard = st.getPrimaryShard("test").name == shard0 ? shard1 : shard0;
assert.commandWorked(mongos.adminCommand({moveChunk: kFullNs, find: {_id: 0}, to: toShard}));

// This insert will fail
db.getCollection(kCollName).insertOne({foo: "foovalue"});

st.stop();

            Assignee:
            robert.sander@mongodb.com Robert Sander
            Reporter:
            pierlauro.sciarelli@mongodb.com Pierlauro Sciarelli
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: