Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16575

intermittent slow inserts with WiredTiger b-tree

XMLWordPrintableJSON

    • Storage Execution
    • ALL
    • Hide

      Run iibench (https://github.com/tmcallaghan/iibench-mongodb) with 2 things changed in run.simple.bash. This uses 10 client threads to load.
      export MAX_ROWS=200000000
      export LENGTH_CHAR_FIELDS=2000

      This problem reproduces for this configuration:
      bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger --wiredTigerEngineConfig="cache_size=32G,direct_io=[data]"

      And I use this mongo.conf:
      processManagement:
      fork: true
      systemLog:
      destination: file
      path: /data/mysql/mongo.28/log
      logAppend: true
      storage:
      dbPath: /data/mysql/mongo.28/data
      journal:
      enabled: true
      mmapv1:
      syncPeriodSecs: 60
      journal:
      commitIntervalMs: 100
      I wasn't able to reproduce it for this configuration:
      bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger

      Show
      Run iibench ( https://github.com/tmcallaghan/iibench-mongodb ) with 2 things changed in run.simple.bash. This uses 10 client threads to load. export MAX_ROWS=200000000 export LENGTH_CHAR_FIELDS=2000 This problem reproduces for this configuration: bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger --wiredTigerEngineConfig="cache_size=32G,direct_io= [data] " And I use this mongo.conf: processManagement: fork: true systemLog: destination: file path: /data/mysql/mongo.28/log logAppend: true storage: dbPath: /data/mysql/mongo.28/data journal: enabled: true mmapv1: syncPeriodSecs: 60 journal: commitIntervalMs: 100 I wasn't able to reproduce it for this configuration: bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger

      I get intermittent slow inserts with the WiredTiger b-tree in 2.8.0rc2. These are the top-10 worst ones from a test in progress. The numbers are milliseconds, so there are many inserts that take more than 100 seconds:
      119597
      118610
      118465
      117977
      117437
      116611
      116498
      116068
      113909
      113342

      This doesn't reproduce with the WiredTiger LSM. I wonder if a slow checkpoint is the problem but I have not done the work to trigger collection of thread stacks when there is a stall, in part because my client code and the error log is printed when the stall ends.

      I will repeat the test with PMP running to get thread stacks every 10 seconds to see if I get stack when the problem occurs.

        1. jmongoiibench.class
          11 kB
          Mark Callaghan
        2. jmongoiibench.java
          35 kB
          Mark Callaghan
        3. jmongoiibench$MyQuery.class
          4 kB
          Mark Callaghan
        4. jmongoiibench$MyReporter.class
          4 kB
          Mark Callaghan
        5. jmongoiibench$MyWriter.class
          4 kB
          Mark Callaghan
        6. performance_output.png
          204 kB
          David Hows
        7. run.simple.bash.L10.Q1
          4 kB
          Mark Callaghan

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            mdcallag Mark Callaghan
            Votes:
            3 Vote for this issue
            Watchers:
            23 Start watching this issue

              Created:
              Updated:
              Resolved: