-
Type:
Bug
-
Resolution: Cannot Reproduce
-
Priority:
Critical - P2
-
None
-
Affects Version/s: WT2.9.3
-
Component/s: None
-
None
-
Environment:OS Name: Microsoft Windows Server 2012 R2 Standard
OS Version: 6.3.9600 N/A Build 9600
System Model: X9DRD-7LN4F(-JBOD)/X9DRD-EF
Processor(s): 2 Processor(s) Installed.
[01]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2100 Mhz
[02]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2100 Mhz
Total Physical Memory: 131 038 MB
Available Physical Memory: 60 664 MBOS Name: Microsoft Windows Server 2012 R2 Standard OS Version: 6.3.9600 N/A Build 9600 System Model: X9DRD-7LN4F(-JBOD)/X9DRD-EF Processor(s): 2 Processor(s) Installed. [01]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2100 Mhz [02]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2100 Mhz Total Physical Memory: 131 038 MB Available Physical Memory: 60 664 MB
I've upgraded wiredtiger v2.9.2 to current version of wiredtiger - v2.9.3 and got deadlock issues.
Usage is simple (just for indexes):
- I have one table with one column - "key"
- Many requests to wiredtiger is binary search on keys range
- Single thread will write or remove keys to the table
- Single thread will make checkpoints and backups on schedule
In about day of application running all is fine. But once wiredtiger starts some checkpoint phase, I got deadlock with v2.9.3.
I tried to inspect callstacks with Sysinternals ProcessExplorer, and got stack like this:
ntoskrnl.exe!KeSynchronizeExecution+0x2246 ntoskrnl.exe!KeWaitForMultipleObjects+0x135e ntoskrnl.exe!KeWaitForMultipleObjects+0xdd9 ntoskrnl.exe!KeWaitForMutexObject+0x373 ntoskrnl.exe!KeStallWhileFrozen+0x1977 ntoskrnl.exe!PoStartNextPowerIrp+0x109d ntoskrnl.exe!KeWaitForMultipleObjects+0x152f ntoskrnl.exe!KeWaitForMultipleObjects+0xdd9 ntoskrnl.exe!FsRtlGetNextBaseMcbEntry+0x327 ntoskrnl.exe!RtlAddAce+0x14a ntoskrnl.exe!setjmpex+0x34a3 ntdll.dll!NtWaitForAlertByThreadId+0xa ntdll.dll!RtlSleepConditionVariableCS+0xc2 KERNELBASE.dll!SleepConditionVariableCS+0x28 WiredTigerNet.dll+0x32e9a WiredTigerNet.dll+0x42472 WiredTigerNet.dll+0x4270f WiredTigerNet.dll+0x446da WiredTigerNet.dll+0x34ebd MSVCR120.dll!beginthreadex+0x107 MSVCR120.dll!endthreadex+0x118 KERNEL32.dll!BaseThreadInitThunk+0x22 ntdll.dll!RtlUserThreadStart+0x34
Then I made minidump of process, and tryed to debug it with Visual Studio, I got following stacktraces:
1 Thread:
[External code: KERNELBASE.dll!SleepConditionVariableCS+0x28] __wt_cond_wait_signal (os_mtx_cond.c:101) __wt_cond_auto_wait_signal (cond_auto.c:63) __wt_evict_thread_run (evict_lru.c:311) __thread_run (thread_group.c:31) [External code: mscrt]
1 Thread:
[External code : KERNELBASE.dll!SleepConditionVariableCS+0x28] __wt_cond_wait_signal (os_mtx_cond.c:101) __sweep_server (conn_sweep.c:278) [External code: mscrt]
1 Thread:
[External code: KERNELBASE.dll!ConditionVariableCS+0x28] __wt_cond_wait (os_mtx_cond.c:101) __wt_readlock (mtx_rw.c:219) __wt_session_lock_dhandle (session_dhandle.c:183) __wt_session_get_btree (session_dhandle.c:506) __conn_btree_apply_internal (conn_dhandle.c:412) __wt_conn_btree_apply (conn_dhandle.c:474) __checkpoint_apply_all (txn_ckpt.c:189) __checkpoint_prepare (txn_ckpt.c:647) __txn_checkpoint (txn_ckpt.c:757) __txn_checkpoint_wrapper (txn_ckpt.c:947) __wt_txn_checkpoint (txn_ckpt.c:1003) __session_checkpoint (session_api.c:1658) __conn_open_session (conn_api.c: 1179) [External code: WiredTigerNet::Session::Checkpoint]
705 Threads:
[External code] __wt_cond_wait_signal (os_mtx_cond.c:101) __wt_readlock (mtx_rw.c:219) __wt_session_lock_dhandle (session_dhandle.c:183) __wt_session_get_btree (session_dhandle.c:506) __wt_session_get_btree_ckpt (session_dhandle.c:346) __wt_curfile_open (cur_file.c:601) __session_open_cursor_int (session_api.c:386) __wt_open_cursor (session_api.c:439) __curtable_open_colgroups (cur_table.c:893) __wt_curtable_open (cur_table.c:1053) __session_open_cursor_int (session_api.c:346) __session_open_cursor (session_api.c:481) [External code: WiredTigerNet::OpenWiredTigerCursor]
WiredTigerNet is simple .NET-wrapper for wiredtiger. I use wiredtiger operations in .NET Threads Pool. When deadlock is occured, thread in pool have blocked and .NET Thread Pool will add new thread to pool every second. Thats the cause of 705 threads are blocked in my case.
For now I get back to v2.9.2. Can you try to fix deadlock with provided stack traces?
- is related to
-
-
- Closed
-
- related to
-
-
- Closed
-