Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20174

WT snapshot threads significantly impact performance

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: WiredTiger
    • None
    • Storage Execution
    • ALL

      Under heavy insert load on a 2-node replica set, WT eviction appears to hang on the secondary.
      Situation was both nodes running on a 24-cpu box, 32GB RAM, pushing 16 threads with vectored writes of 100 tiny documents at a time

      Per Michael, this seems to be related to a snapshot pinning down the memory

      -  24.71%  mongod  mongod               [.] __rec_txn_read                                        
         - __rec_txn_read                                                                                
            - 99.65% __rec_row_leaf_insert                                                              
                 __wt_reconcile                                                                          
                 __wt_evict                                                                              
                 __wt_evict_page                                                                        
                 __wt_page_in_func                                                                      
                 __wt_row_search                                                                        
                 __wt_btcur_insert                                                                      
                 __curfile_insert                                                                        
                 mongo::WiredTigerIndexUnique::_insert(__wt_cursor*, mongo::BSONObj const&, mongo::Record
                 mongo::WiredTigerIndex::insert(mongo::OperationContext*, mongo::BSONObj const&, mongo::R
                 mongo::IndexAccessMethod::insert(mongo::OperationContext*, mongo::BSONObj const&, mongo:
                 mongo::IndexCatalog::_indexFilteredRecords(mongo::OperationContext*, mongo::IndexCatalog
                 mongo::IndexCatalog::_indexRecords(mongo::OperationContext*, mongo::IndexCatalogEntry*,
                 mongo::IndexCatalog::indexRecords(mongo::OperationContext*, std::vector<mongo::BsonRecor
                 mongo::Collection::_insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterat
                 mongo::Collection::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterato
                 mongo::Collection::insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::applyOperation_inlock(mongo::OperationContext*, mongo::Database*, mongo::BS
                 std::_Function_handler<mongo::Status (mongo::OperationContext*, mongo::Database*, mongo:
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool)
                 mongo::repl::multiSyncApply(std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >
                 mongo::ThreadPool::_doOneTask(std::unique_lock<std::mutex>*)                            
                 mongo::ThreadPool::_consumeTasks()                                                      
                 mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&)            
                 0x7f957762ea40                                                                          
      -  22.88%  mongod  mongod               [.] __wt_row_modify                                        
         - __wt_row_modify                                                                              
            - 99.42% __split_multi_inmem                                                                
                 __wt_split_rewrite                                                                      
                 __wt_evict                                                                              
                 __wt_evict_page                                                                        
                 __wt_page_in_func                                                                      
                 __wt_row_search                                                                        
                 __wt_btcur_insert                                                                      
                 __curfile_insert                                                                        
                 mongo::WiredTigerIndexUnique::_insert(__wt_cursor*, mongo::BSONObj const&, mongo::Record
                 mongo::WiredTigerIndex::insert(mongo::OperationContext*, mongo::BSONObj const&, mongo::R
                 mongo::IndexAccessMethod::insert(mongo::OperationContext*, mongo::BSONObj const&, mongo:
                 mongo::IndexCatalog::_indexFilteredRecords(mongo::OperationContext*, mongo::IndexCatalog
                 mongo::IndexCatalog::_indexRecords(mongo::OperationContext*, mongo::IndexCatalogEntry*,
                 mongo::IndexCatalog::indexRecords(mongo::OperationContext*, std::vector<mongo::BsonRecor
                 mongo::Collection::_insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterat
                 mongo::Collection::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterato
                 mongo::Collection::insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::applyOperation_inlock(mongo::OperationContext*, mongo::Database*, mongo::BS
                 std::_Function_handler<mongo::Status (mongo::OperationContext*, mongo::Database*, mongo:
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool)
                 mongo::repl::multiSyncApply(std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >
                 mongo::ThreadPool::_doOneTask(std::unique_lock<std::mutex>*)                            
                 mongo::ThreadPool::_consumeTasks()                                                      
                 mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&)            
                 0x7f957762ea40                                                                          
            + 0.58% __wt_split_rewrite                                                                  
      -  21.43%  mongod  mongod               [.] __wt_row_search                                        
         - __wt_row_search                                                                              
            - 99.71% __split_multi_inmem                                                                
                 __wt_split_rewrite                                                                      
                 __wt_evict                                                                              
                 __wt_evict_page                                                                        
                 __wt_page_in_func                                                                      
                 __wt_row_search                                                                        
                 __wt_btcur_insert                                                                      
                 __curfile_insert                                                                        
                 mongo::WiredTigerIndexUnique::_insert(__wt_cursor*, mongo::BSONObj const&, mongo::Record
                 mongo::WiredTigerIndex::insert(mongo::OperationContext*, mongo::BSONObj const&, mongo::R
                 mongo::IndexAccessMethod::insert(mongo::OperationContext*, mongo::BSONObj const&, mongo:
                 mongo::IndexCatalog::_indexFilteredRecords(mongo::OperationContext*, mongo::IndexCatalog
                 mongo::IndexCatalog::_indexRecords(mongo::OperationContext*, mongo::IndexCatalogEntry*,
                 mongo::IndexCatalog::indexRecords(mongo::OperationContext*, std::vector<mongo::BsonRecor
                 mongo::Collection::_insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterat
                 mongo::Collection::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterato
                 mongo::Collection::insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::applyOperation_inlock(mongo::OperationContext*, mongo::Database*, mongo::BS
                 std::_Function_handler<mongo::Status (mongo::OperationContext*, mongo::Database*, mongo:
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool,
                 mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, bool)
                 mongo::repl::multiSyncApply(std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >
                 mongo::ThreadPool::_doOneTask(std::unique_lock<std::mutex>*)                            
                 mongo::ThreadPool::_consumeTasks()                                                      
                 mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&)    
      

        1. benchRun
          1 kB
        2. benchSet
          0.6 kB
        3. md_rs2
          0.6 kB
        4. md_stop
          0.1 kB
        5. pymongo_init_rs
          0.7 kB

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            martin.bligh Martin Bligh
            Votes:
            1 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: