wtperf crashed last night with a dhandle locking race, during close (running evict-lsm).
The stacks are simple:
(gdb) thread 2 [Switching to thread 2 (Thread 802007400 (LWP 100228/wtperf))]#0 0x0000000800d1a87c in pthread_kill () from /lib/libthr.so.3 (gdb) where #0 0x0000000800d1a87c in pthread_kill () from /lib/libthr.so.3 WT-1 0x0000000800d1b8c9 in pthread_kill () from /lib/libthr.so.3 WT-2 0x00000000004166c5 in __wt_thread_join (session=0x80204b000, tid=0x802008800) at ../src/os_posix/os_thread.c:35 WT-3 0x000000000040c9b8 in __wt_connection_close (conn=0x80201fb00) at ../src/conn/conn_open.c:118 WT-4 0x0000000000408c37 in __conn_close (wt_conn=0x80201fb00, config=0x0) at ../src/conn/conn_api.c:518 WT-5 0x00000000004053dd in main (argc=5, argv=0x7fffffffd9a0) at ../../../bench/wtperf/wtperf.c:1064 (gdb) thread 1 [Switching to thread 1 (Thread 802008800 (LWP 104683/wtperf))]#0 0x00000000004311e1 in __evict_walk (session=0x80204b210, entriesp=0x7fffff7fbed4, clean=1) at ../src/btree/bt_evict.c:722 722 if (!WT_PREFIX_MATCH(dhandle->name, "file:") || (gdb) where #0 0x00000000004311e1 in __evict_walk (session=0x80204b210, entriesp=0x7fffff7fbed4, clean=1) at ../src/btree/bt_evict.c:722 WT-1 0x0000000000430f09 in __evict_lru (session=0x80204b210, clean=1) at ../src/btree/bt_evict.c:631 WT-2 0x0000000000430243 in __evict_worker (session=0x80204b210) at ../src/btree/bt_evict.c:252 WT-3 0x000000000042fe93 in __wt_cache_evict_server (arg=0x80204b210) at ../src/btree/bt_evict.c:160 WT-4 0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3 WT-5 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff7fc000 (gdb) p dhandle $1 = (WT_DATA_HANDLE *) 0xabababababababab
In short, close is discarding handles, and the eviction server is still walking them.