-
Type: Improvement
-
Resolution: Done
-
Priority: Minor - P4
-
Affects Version/s: None
-
Component/s: WiredTiger
-
None
-
Fully Compatible
I noticed that the repl13.js test is very slow with WiredTiger and LSM. While digging into the performance, I saw that the slow part of the test was building an index on the secondary.
It appears as though each entry is inserted with a WiredTiger auto-commit transaction, each auto-commit transaction is doing a write and fsync to the WiredTiger WAL (journal). The index being built is 100,000 entries - so 100k fsync calls.
A representative call stack is:
(gdb) where #0 0x00007fecbb6d8c4d in fdatasync () from /lib64/libc.so.6 #1 0x0000000001336e7c in __wt_handle_sync (fd=18) at src/third_party/wiredtiger/src/os_posix/os_fsync.c:44 #2 0x0000000001337192 in __wt_fsync (session=session@entry=0x3a8a040, fh=0x3657750) at src/third_party/wiredtiger/src/os_posix/os_fsync.c:129 #3 0x0000000001321a1a in __log_release (session=session@entry=0x3a8a040, slot=slot@entry=0x7fecae8c5a00, freep=freep@entry=0x7fecae8c59ec) at src/third_party/wiredtiger/src/log/log.c:1043 #4 0x0000000001322b28 in __log_direct_write (session=session@entry=0x3a8a040, record=record@entry=0x3a3f410, lsnp=lsnp@entry=0x0, flags=flags@entry=4) at src/third_party/wiredtiger/src/log/log.c:1542 #5 0x0000000001324139 in __log_write_internal (flags=4, lsnp=0x0, record=0x3a3f410, session=0x3a8a040) at src/third_party/wiredtiger/src/log/log.c:1707 #6 __wt_log_write (session=session@entry=0x3a8a040, record=<optimized out>, lsnp=lsnp@entry=0x0, flags=4) at src/third_party/wiredtiger/src/log/log.c:1649 #7 0x000000000136ca84 in __wt_txn_log_commit (session=session@entry=0x3a8a040, cfg=cfg@entry=0x0) at src/third_party/wiredtiger/src/txn/txn_log.c:209 #8 0x0000000001369b10 in __wt_txn_commit (session=0x3a8a040, cfg=0x0) at src/third_party/wiredtiger/src/txn/txn.c:419 #9 0x000000000132b785 in __clsm_insert (cursor=0x3915ee0) at src/third_party/wiredtiger/src/lsm/lsm_cursor.c:1361 #10 0x0000000000d71298 in mongo::WiredTigerIndex::UniqueBulkBuilder::doInsert (this=0x50d6340) at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:582 #11 0x0000000000d71945 in mongo::WiredTigerIndex::UniqueBulkBuilder::addKey (this=0x50d6340, newKey=..., loc=...) at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:531 #12 0x0000000000a92fb7 in mongo::IndexAccessMethod::commitBulk (this=0x5b1bd40, txn=0x7fecae8c78f0, bulk=std::unique_ptr<mongo::IndexAccessMethod::BulkBuilder> containing 0x5b11e80, mayInterrupt=false, dupsAllowed=false, dupsToDrop=dupsToDrop@entry=0x7fecae8c6810) at src/mongo/db/index/index_access_method.cpp:410 #13 0x0000000000949b28 in mongo::MultiIndexBlock::doneInserting (this=this@entry=0x7fecae8c6840, dupsOut=dupsOut@entry=0x7fecae8c6810) at src/mongo/db/catalog/index_create.cpp:371 #14 0x000000000094a130 in mongo::MultiIndexBlock::insertAllDocumentsInCollection ( this=this@entry=0x7fecae8c6840, dupsOut=dupsOut@entry=0x7fecae8c6810) at src/mongo/db/catalog/index_create.cpp:328 #15 0x0000000000955b13 in mongo::Cloner::go (this=this@entry=0x7fecae8c6940, txn=txn@entry=0x7fecae8c78f0, toDBName="d", masterHost="127.0.0.1:31000", opts=..., clonedColls=clonedColls@entry=0x0, errmsg="", errCode=errCode@entry=0x7fecae8c690c) at src/mongo/db/cloner.cpp:628 #16 0x0000000000c0cf46 in mongo::repl::ReplSource::resync (this=this@entry=0x3a0e3c0, txn=txn@entry=0x7fecae8c78f0, dbName=...) at src/mongo/db/repl/master_slave.cpp:492 #17 0x0000000000c108c8 in mongo::repl::ReplSource::_sync_pullOpLog_applyOperation ( this=this@entry=0x3a0e3c0, txn=txn@entry=0x7fecae8c78f0, op=..., alreadyLocked=alreadyLocked@entry=false) at src/mongo/db/repl/master_slave.cpp:765
Ideally doing such an index build in WiredTiger would turn on bulk loading, or wrap groups of transactions into commits or in the least doing non-sync transactional operations.
The obvious place to make that change is in mongo::MultiIndexBlock::insertAllDocumentsInCollection, but that's outside the WiredTiger storage engine implementation layer, so I'm hesitant to jump into the change without some discussion.
There is already a call to WiredTigerRecoveryUnit::setRollbackWritesDisabled in IndexAccessMethod::commitBulk, but it isn't obvious how to use that information to help from the WiredTiger storage engine side.
- depends on
-
WT-1922 Implement bulk load for LSM tree
- Closed