Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8550

Bump the maximum number of opened hs cursors to 3

    • 8
    • Storage - Ra 2022-01-10

      A given session in WiredTiger had it's hs_cursor_counter exceed the allowed maximum of 2 which causes the assertion in api.h:92 to fire. WT_ASSERT(s, (s)>api_call_counter > 1 || (s)>hs_cursor_counter <= 2);.

      This occurred during the checkpoint call from a clean mongodb shutdown. Callstack below:

      (gdb) bt
      #0  0x00007f0aa046efb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
      #1  0x00007f0aa0470921 in abort () from /lib/x86_64-linux-gnu/libc.so.6
      #2  0x000055c9581f1928 in __wt_abort (session=<optimized out>) at src/third_party/wiredtiger/src/os_common/os_abort.c:30
      #3  0x000055c95813e349 in __wt_cursor_get_valuev (cursor=<optimized out>, ap=0x7f0a989aeb30) at src/third_party/wiredtiger/src/cursor/cur_std.c:547
      #4  0x000055c95813d857 in __wt_cursor_get_value (cursor=0x2) at src/third_party/wiredtiger/src/cursor/cur_std.c:507
      #5  0x000055c958254192 in __hs_delete_reinsert_from_pos (session=<optimized out>, hs_cursor=0x7f0a804fbc20, btree_id=<optimized out>, key=<optimized out>, ts=<optimized out>, reinsert=<optimized out>, error_on_ooo_ts=<optimized out>, counter=0x7f0a989aee30) at src/third_party/wiredtiger/src/history/hs_rec.c:1028
      #6  0x000055c95825321d in __hs_insert_record (session=0x7f0a9412b830, cursor=<optimized out>, btree=<optimized out>, key=0x7f0a9171bda0, type=3 '\003', hs_value=0x7f0a90e076a0, tw=0x7f0a989aef60, error_on_ooo_ts=<optimized out>) at src/third_party/wiredtiger/src/history/hs_rec.c:219
      #7  0x000055c95824f992 in __wt_hs_insert_updates (session=<optimized out>, r=<optimized out>, multi=<optimized out>) at src/third_party/wiredtiger/src/history/hs_rec.c:712
      #8  0x000055c958241708 in __rec_hs_wrapup (session=<optimized out>, r=<optimized out>) at src/third_party/wiredtiger/src/reconcile/rec_write.c:2507
      #9  __rec_write_wrapup (session=0x8, r=<optimized out>, page=0x7f0a9a736d40) at src/third_party/wiredtiger/src/reconcile/rec_write.c:2284
      #10 0x000055c9582386bc in __reconcile (session=<optimized out>, ref=<optimized out>, salvage=<optimized out>, flags=<optimized out>, page_lockedp=0x61af638301) at src/third_party/wiredtiger/src/reconcile/rec_write.c:309
      #11 __wt_reconcile (session=0x7f0a9412b830, ref=<optimized out>, salvage=0x0, flags=<optimized out>) at src/third_party/wiredtiger/src/reconcile/rec_write.c:97
      #12 0x000055c9581d7f91 in __wt_sync_file (session=<optimized out>, syncop=<optimized out>) at src/third_party/wiredtiger/src/btree/bt_sync.c:658
      #13 0x000055c958344a39 in __checkpoint_tree (session=<optimized out>, is_checkpoint=<optimized out>, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1880
      #14 0x000055c95834a693 in __checkpoint_tree_helper (session=0x7f0a9412b830, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1996
      #15 0x000055c95834a55d in __checkpoint_apply_to_dhandles (session=0x7f0a9412b830, cfg=0x7f0a989affd0, op=0x55c95834a5e0 <__checkpoint_tree_helper>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:198
      #16 0x000055c958347199 in __txn_checkpoint (session=0x7f0a9412b830, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:922
      #17 0x000055c9583435b1 in __txn_checkpoint_wrapper (session=0x7f0a9412b830, cfg=0x7f0a989ae7a0) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1165
      #18 0x000055c958343147 in __wt_txn_checkpoint (session=0x7f0a9412b830, cfg=0x7f0a989affd0, waiting=true) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1219
      #19 0x000055c95833e74c in __wt_txn_global_shutdown (session=0x7f0a9412b020, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn.c:2595
      #20 0x000055c958017be3 in __conn_close (wt_conn=<optimized out>, config=<optimized out>) at src/third_party/wiredtiger/src/conn/conn_api.c:1175
      #21 0x000055c957f444db in mongo::WiredTigerKVEngine::cleanShutdown (this=0x7f0a98b51820) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:830
      #22 0x000055c95899c8b6 in mongo::StorageEngineImpl::cleanShutdown (this=<optimized out>) at src/mongo/db/storage/storage_engine_impl.cpp:739
      #23 0x000055c9589bf486 in mongo::(anonymous namespace)::shutdownGlobalStorageEngineCleanly (service=0x7f0a9a6a8a20, errorToReport=...) at src/mongo/db/storage/storage_engine_init.cpp:204
      #24 0x000055c9589bf346 in mongo::shutdownGlobalStorageEngineCleanly (service=0x2) at src/mongo/db/storage/storage_engine_init.cpp:214
      #25 0x000055c956b2ec94 in mongo::(anonymous namespace)::shutdownTask (shutdownArgs=...) at src/mongo/db/mongod_main.cpp:1432
      #26 0x000055c95b213331 in mongo::(anonymous namespace)::runTasks(std::stack<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)>, std::deque<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)>, std::allocator<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)> > > >, mongo::ShutdownTaskArgs const&) (tasks=std::stack wrapping: std::deque with 1 element = {...}, shutdownArgs=...) at src/mongo/util/exit.cpp:61
      #27 0x000055c95b2130a6 in mongo::shutdown (code=mongo::ExitCode::EXIT_CLEAN, shutdownArgs=...) at src/mongo/util/exit.cpp:141
      #28 0x000055c956b44371 in mongo::exitCleanly (code=mongo::ExitCode::EXIT_BADOPTIONS) at src/mongo/util/exit.h:104
      #29 0x000055c9586e5e22 in mongo::(anonymous namespace)::handleOneSignal (waited=..., rotation=<optimized out>) at src/mongo/util/signal_handlers.cpp:263
      #30 0x000055c9586e573b in mongo::(anonymous namespace)::signalProcessingThread (rotate=<optimized out>) at src/mongo/util/signal_handlers.cpp:304
      #31 0x000055c9586e8376 in std::__invoke_impl<void, void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__f=<optimized out>, __args=@0x7f0a9a820708: mongo::LogFileStatus::kNeedToRotateLogFile) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60
      #32 std::__invoke<void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__fn=<optimized out>, __args=@0x7f0a9a820708: mongo::LogFileStatus::kNeedToRotateLogFile) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95
      #33 std::__apply_impl<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus>, 0ul> (__f=<optimized out>, __t=...) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1678
      #34 std::apply<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus> > (__f=<optimized out>, __t=...) at /opt/mongodbtoolchain/revisions/8391ec859b03d3bebb666821a88e616ae61a567a/stow/gcc-v3.exV/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1687
      #35 mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}::operator()() (this=<optimized out>) at src/mongo/stdx/thread.h:193
      #36 0x000055c95b4436df in execute_native_thread_routine ()
      #37 0x00007f0aa0a406db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #38 0x00007f0aa055171f in clone () from /lib/x86_64-linux-gnu/libc.so.6
      

            Assignee:
            etienne.petrel@mongodb.com Etienne Petrel
            Reporter:
            luke.pearson@mongodb.com Luke Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: