Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-89841

Investigate DDL LockBusy errors in config transition suite tests

    • Type: Icon: Task Task
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • CAR Team 2024-05-13, CAR Team 2024-05-27

      SERVER-89182 adds a concurrency suite with random migrations and config shard transitions, and several tests can fail with LockBusy errors taking DDL locks for DDL operations or in the background metadata consistency checker. This seems most likely on slower variants, but not exclusively on them, and seems to only involve tests that do many DDL operations.

      This ticket tracks diagnosing this failure and either fixing it or officially disabling the affected tests in the config shard transition suite.

      The disabled tests:

      • jstests/concurrency/fsm_workloads/agg_sort.js
      • jstests/concurrency/fsm_workloads/map_reduce_interrupt.js
      • jstests/concurrency/fsm_workloads/map_reduce_replace.js
      • jstests/concurrency/fsm_workloads/map_reduce_replace_remove.js
      • jstests/concurrency/fsm_workloads/map_reduce_replace_nonexistent.js
      • jstests/concurrency/fsm_workloads/random_DDL_operations.js
      • jstests/concurrency/fsm_workloads/rename_sharded_collection.js

      Example failure:

      2024-04-23T22:51:12.605Z assert: command failed: {
      "ok" : 0,
      "errmsg" : "Failed to acquire DDL lock for 'test30_fsmdb0' in mode S after 300000 ms that is currently locked by '{\"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.dfdbc260-8704-4fe2-8936-2723c8937651\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.dc87d861-1937-4991-8f53-16cc5375b11e\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.ea14da7e-1710-444b-9420-4fb390ed055e\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.9ade3ec4-31f3-4733-82a6-1c2cfe1fdc09\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.88e82a10-0b4d-4dc3-8b33-f2d8874e07fa\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.map_reduce_replace3\", operationType: \"createCollection_V4\" }\", lock mode: \"IX\" }\"}'",
      "code" : 46,
      "codeName" : "LockBusy",
      "$clusterTime" : {
      "clusterTime" : Timestamp(1713912672, 20),
      "signature" : {
      "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
      "keyId" : NumberLong(0)
      }
      },
      "operationTime" : Timestamp(1713912672, 20)
      } with original command request: {
      "checkMetadataConsistency" : 1,
      "lsid" : {
      "id" : UUID("acceb5ff-7ffc-468a-8ea7-ecd7871a1b8f")
      },
      "$clusterTime" : {
      "clusterTime" : Timestamp(1713912372, 34),
      "signature" : {
      "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
      "keyId" : NumberLong(0)
      }
      }
      } on connection: connection to localhost:20009,localhost:20010
      _getErrorWithCode@src/mongo/shell/utils.js:24:13
      doassert@src/mongo/shell/assert.js:18:14
      _assertCommandWorked@src/mongo/shell/assert.js:761:25
      assert.commandWorked@src/mongo/shell/assert.js:855:16
      DB.prototype.checkMetadataConsistency@src/mongo/shell/db.js:1727:24
      checkMetadataConsistency@/data/mci/99663d3669209ec618205300ac878a78/src/jstests/libs/check_metadata_consistency_helpers.js:52:45
      run@/data/mci/99663d3669209ec618205300ac878a78/src/jstests/libs/check_metadata_consistency_helpers.js:61:13
      @jstests/hooks/run_check_metadata_consistency.js:33:28 

            Assignee:
            pierlauro.sciarelli@mongodb.com Pierlauro Sciarelli
            Reporter:
            jack.mulrow@mongodb.com Jack Mulrow
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: