-
Type: Bug
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: None
-
8
-
Storage - Ra 2022-01-10
A given session in WiredTiger had it's hs_cursor_counter exceed the allowed maximum of 2 which causes the assertion in api.h:92 to fire. WT_ASSERT(s, (s)>api_call_counter > 1 || (s)>hs_cursor_counter <= 2);.
This occurred during the checkpoint call from a clean mongodb shutdown. Callstack below:
(gdb) bt #0 0x00007f0aa046efb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f0aa0470921 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x000055c9581f1928 in __wt_abort (session=<optimized out>) at src/third_party/wiredtiger/src/os_common/os_abort.c:30 #3 0x000055c95813e349 in __wt_cursor_get_valuev (cursor=<optimized out>, ap=0x7f0a989aeb30) at src/third_party/wiredtiger/src/cursor/cur_std.c:547 #4 0x000055c95813d857 in __wt_cursor_get_value (cursor=0x2) at src/third_party/wiredtiger/src/cursor/cur_std.c:507 #5 0x000055c958254192 in __hs_delete_reinsert_from_pos (session=<optimized out>, hs_cursor=0x7f0a804fbc20, btree_id=<optimized out>, key=<optimized out>, ts=<optimized out>, reinsert=<optimized out>, error_on_ooo_ts=<optimized out>, counter=0x7f0a989aee30) at src/third_party/wiredtiger/src/history/hs_rec.c:1028 #6 0x000055c95825321d in __hs_insert_record (session=0x7f0a9412b830, cursor=<optimized out>, btree=<optimized out>, key=0x7f0a9171bda0, type=3 '\003', hs_value=0x7f0a90e076a0, tw=0x7f0a989aef60, error_on_ooo_ts=<optimized out>) at src/third_party/wiredtiger/src/history/hs_rec.c:219 #7 0x000055c95824f992 in __wt_hs_insert_updates (session=<optimized out>, r=<optimized out>, multi=<optimized out>) at src/third_party/wiredtiger/src/history/hs_rec.c:712 #8 0x000055c958241708 in __rec_hs_wrapup (session=<optimized out>, r=<optimized out>) at src/third_party/wiredtiger/src/reconcile/rec_write.c:2507 #9 __rec_write_wrapup (session=0x8, r=<optimized out>, page=0x7f0a9a736d40) at src/third_party/wiredtiger/src/reconcile/rec_write.c:2284 #10 0x000055c9582386bc in __reconcile (session=<optimized out>, ref=<optimized out>, salvage=<optimized out>, flags=<optimized out>, page_lockedp=0x61af638301) at src/third_party/wiredtiger/src/reconcile/rec_write.c:309 #11 __wt_reconcile (session=0x7f0a9412b830, ref=<optimized out>, salvage=0x0, flags=<optimized out>) at src/third_party/wiredtiger/src/reconcile/rec_write.c:97 #12 0x000055c9581d7f91 in __wt_sync_file (session=<optimized out>, syncop=<optimized out>) at src/third_party/wiredtiger/src/btree/bt_sync.c:658 #13 0x000055c958344a39 in __checkpoint_tree (session=<optimized out>, is_checkpoint=<optimized out>, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1880 #14 0x000055c95834a693 in __checkpoint_tree_helper (session=0x7f0a9412b830, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1996 #15 0x000055c95834a55d in __checkpoint_apply_to_dhandles (session=0x7f0a9412b830, cfg=0x7f0a989affd0, op=0x55c95834a5e0 <__checkpoint_tree_helper>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:198 #16 0x000055c958347199 in __txn_checkpoint (session=0x7f0a9412b830, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:922 #17 0x000055c9583435b1 in __txn_checkpoint_wrapper (session=0x7f0a9412b830, cfg=0x7f0a989ae7a0) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1165 #18 0x000055c958343147 in __wt_txn_checkpoint (session=0x7f0a9412b830, cfg=0x7f0a989affd0, waiting=true) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1219 #19 0x000055c95833e74c in __wt_txn_global_shutdown (session=0x7f0a9412b020, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn.c:2595 #20 0x000055c958017be3 in __conn_close (wt_conn=<optimized out>, config=<optimized out>) at src/third_party/wiredtiger/src/conn/conn_api.c:1175 #21 0x000055c957f444db in mongo::WiredTigerKVEngine::cleanShutdown (this=0x7f0a98b51820) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:830 #22 0x000055c95899c8b6 in mongo::StorageEngineImpl::cleanShutdown (this=<optimized out>) at src/mongo/db/storage/storage_engine_impl.cpp:739 #23 0x000055c9589bf486 in mongo::(anonymous namespace)::shutdownGlobalStorageEngineCleanly (service=0x7f0a9a6a8a20, errorToReport=...) at src/mongo/db/storage/storage_engine_init.cpp:204 #24 0x000055c9589bf346 in mongo::shutdownGlobalStorageEngineCleanly (service=0x2) at src/mongo/db/storage/storage_engine_init.cpp:214 #25 0x000055c956b2ec94 in mongo::(anonymous namespace)::shutdownTask (shutdownArgs=...) at src/mongo/db/mongod_main.cpp:1432 #26 0x000055c95b213331 in mongo::(anonymous namespace)::runTasks(std::stack<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)>, std::deque<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)>, std::allocator<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)> > > >, mongo::ShutdownTaskArgs const&) (tasks=std::stack wrapping: std::deque with 1 element = {...}, shutdownArgs=...) at src/mongo/util/exit.cpp:61 #27 0x000055c95b2130a6 in mongo::shutdown (code=mongo::ExitCode::EXIT_CLEAN, shutdownArgs=...) at src/mongo/util/exit.cpp:141 #28 0x000055c956b44371 in mongo::exitCleanly (code=mongo::ExitCode::EXIT_BADOPTIONS) at src/mongo/util/exit.h:104 #29 0x000055c9586e5e22 in mongo::(anonymous namespace)::handleOneSignal (waited=..., rotation=<optimized out>) at src/mongo/util/signal_handlers.cpp:263 #30 0x000055c9586e573b in mongo::(anonymous namespace)::signalProcessingThread (rotate=<optimized out>) at src/mongo/util/signal_handlers.cpp:304 #31 0x000055c9586e8376 in std::__invoke_impl<void, void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__f=<optimized out>, __args=@0x7f0a9a820708: mongo::LogFileStatus::kNeedToRotateLogFile) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 #32 std::__invoke<void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__fn=<optimized out>, __args=@0x7f0a9a820708: mongo::LogFileStatus::kNeedToRotateLogFile) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 #33 std::__apply_impl<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus>, 0ul> (__f=<optimized out>, __t=...) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1678 #34 std::apply<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus> > (__f=<optimized out>, __t=...) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1687 #35 mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}::operator()() (this=<optimized out>) at src/mongo/stdx/thread.h:193 #36 0x000055c95b4436df in execute_native_thread_routine () #37 0x00007f0aa0a406db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #38 0x00007f0aa055171f in clone () from /lib/x86_64-linux-gnu/libc.so.6
- is caused by
-
WT-8476 Some refactorings now that FLCS is less of a special case
- Closed