-
Type: Task
-
Resolution: Gone away
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Catalog and Routing
-
CAR Team 2024-05-13, CAR Team 2024-05-27
SERVER-89182 adds a concurrency suite with random migrations and config shard transitions, and several tests can fail with LockBusy errors taking DDL locks for DDL operations or in the background metadata consistency checker. This seems most likely on slower variants, but not exclusively on them, and seems to only involve tests that do many DDL operations.
This ticket tracks diagnosing this failure and either fixing it or officially disabling the affected tests in the config shard transition suite.
The disabled tests:
- jstests/concurrency/fsm_workloads/agg_sort.js
- jstests/concurrency/fsm_workloads/map_reduce_interrupt.js
- jstests/concurrency/fsm_workloads/map_reduce_replace.js
- jstests/concurrency/fsm_workloads/map_reduce_replace_remove.js
- jstests/concurrency/fsm_workloads/map_reduce_replace_nonexistent.js
- jstests/concurrency/fsm_workloads/random_DDL_operations.js
- jstests/concurrency/fsm_workloads/rename_sharded_collection.js
Example failure:
2024-04-23T22:51:12.605Z assert: command failed: { "ok" : 0, "errmsg" : "Failed to acquire DDL lock for 'test30_fsmdb0' in mode S after 300000 ms that is currently locked by '{\"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.dfdbc260-8704-4fe2-8936-2723c8937651\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.dc87d861-1937-4991-8f53-16cc5375b11e\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.ea14da7e-1710-444b-9420-4fb390ed055e\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.9ade3ec4-31f3-4733-82a6-1c2cfe1fdc09\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.tmp.agg_out.88e82a10-0b4d-4dc3-8b33-f2d8874e07fa\", operationType: \"renameCollection_V2\" }\", lock mode: \"IX\" }\", \"{ operation: \"{ namespace: \"test30_fsmdb0.map_reduce_replace3\", operationType: \"createCollection_V4\" }\", lock mode: \"IX\" }\"}'", "code" : 46, "codeName" : "LockBusy", "$clusterTime" : { "clusterTime" : Timestamp(1713912672, 20), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1713912672, 20) } with original command request: { "checkMetadataConsistency" : 1, "lsid" : { "id" : UUID("acceb5ff-7ffc-468a-8ea7-ecd7871a1b8f") }, "$clusterTime" : { "clusterTime" : Timestamp(1713912372, 34), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } on connection: connection to localhost:20009,localhost:20010 _getErrorWithCode@src/mongo/shell/utils.js:24:13 doassert@src/mongo/shell/assert.js:18:14 _assertCommandWorked@src/mongo/shell/assert.js:761:25 assert.commandWorked@src/mongo/shell/assert.js:855:16 DB.prototype.checkMetadataConsistency@src/mongo/shell/db.js:1727:24 checkMetadataConsistency@/data/mci/99663d3669209ec618205300ac878a78/src/jstests/libs/check_metadata_consistency_helpers.js:52:45 run@/data/mci/99663d3669209ec618205300ac878a78/src/jstests/libs/check_metadata_consistency_helpers.js:61:13 @jstests/hooks/run_check_metadata_consistency.js:33:28
- is related to
-
SERVER-56879 Make DDLs operations resilient to add/remove shard
- Closed
- related to
-
SERVER-88188 Check metadata consistency hook should not fail on LockBusy error
- Closed
-
SERVER-91082 Complete TODO listed in SERVER-89841
- Backlog
-
SERVER-90786 SERVER-90786 execute again tests excluded due to add/remove shard issue solved by SERVER-56879
- Closed