Critical - P2 Doesn't impact clusters that have the _SecondaryThrottle turned off or have the balancer off.
MigrationBatchFetcher's destructor depends on calling inserterWorkerThreadPool shutdown/join to drain all the scheduled jobs and join on the threadPool spawns another extra thread to drain it.
Also, secondaryThrottle is disabled for a chunkMigrationConcurrency of more than 1 because of how sessions are checked out/in.
When the MigrationBatchFetcher calls join on the inserterThreadPool and there are more jobs to process in the pool, it can spawn an extra thread to drain the pool. So now, although the threadCnt is 1 here, 2 threads are competing for secondaryThrottle and the invariant is caused.
Invariant failure string in the log file looked like -
| {"t":\{"$date":"2023-02-28T19:58:58.693+00:00"}
,"s":"F", "c":"-", "id":23079, "ctx":"ChunkMigrationInserters-1","msg":"Invariant failure","attr":{"expr":"checkedOutSession","file":"src/mongo/db/session_catalog.cpp","line":302}} |
Workaround
Either one of the following would work
1. Turn the balancer off
2. Turn the secondaryThrottle off(with the balancer still running).