Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16717

unexpected WriteConflict exceptions with WiredTiger b-tree

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.0-rc6
    • Affects Version/s: 2.8.0-rc4
    • Component/s: Storage
    • Fully Compatible
    • ALL
    • Hide

      Start mongod:
      numactl --interleave=all bin/mongod --config /data/mysql/mongo.280rc4/mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 2

      Run iibench (https://github.com/tmcallaghan/iibench-mongodb) with a few changes to run.simple.bash:
      export MAX_ROWS=400000000
      export LENGTH_CHAR_FIELDS=2000

      I also edited jmongoiibench.java to increase the client timeout to avoid errors on checkpoint stalls.

      I used this mongo.conf:
      processManagement:
      fork: true
      systemLog:
      destination: file
      path: /data/mysql/mongo.280rc4/log
      logAppend: true
      storage:
      dbPath: /data/mysql/mongo.280rc4/data
      journal:
      enabled: true
      mmapv1:
      syncPeriodSecs: 60
      journal:
      commitIntervalMs: 100
      storage.wiredTiger.collectionConfig.blockCompressor: zlib
      storage.wiredTiger.engineConfig.journalCompressor: snappy
      operationProfiling.slowOpThresholdMs: 2000

      Show
      Start mongod: numactl --interleave=all bin/mongod --config /data/mysql/mongo.280rc4/mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 2 Run iibench ( https://github.com/tmcallaghan/iibench-mongodb ) with a few changes to run.simple.bash: export MAX_ROWS=400000000 export LENGTH_CHAR_FIELDS=2000 I also edited jmongoiibench.java to increase the client timeout to avoid errors on checkpoint stalls. I used this mongo.conf: processManagement: fork: true systemLog: destination: file path: /data/mysql/mongo.280rc4/log logAppend: true storage: dbPath: /data/mysql/mongo.280rc4/data journal: enabled: true mmapv1: syncPeriodSecs: 60 journal: commitIntervalMs: 100 storage.wiredTiger.collectionConfig.blockCompressor: zlib storage.wiredTiger.engineConfig.journalCompressor: snappy operationProfiling.slowOpThresholdMs: 2000

      This occurs when running iibench with the WiredTiger b-tree. This does not occur with mmapv1, wiredtiger LSM, tokumx or rocksdb engines for MongoDB. While I won't rule out the client code, it looks like a WT bug to me.

      It seems to occur around the time of a checkpoint stall.

      The iibench client prints this on an error:
      com.mongodb.WriteConcernException:

      { "serverUsed" : "localhost:27017" , "ok" : 1 , "n" : 0 , "err" : "WriteConflict" , "code" : 112}

      And the mongod error log has:
      2015-01-04T21:49:45.625-0800 I QUERY [conn10] command iibench.$cmd command: insert

      { insert: "purchases_index", ordered: true, documents: 1000 }

      keyUpdates:0 reslen:40 12929ms
      2015-01-04T21:49:45.654-0800 I QUERY [conn4] command iibench.$cmd command: insert

      { insert: "purchases_index", ordered: true, documents: 1000 }

      keyUpdates:0 reslen:40 12894ms
      2015-01-04T21:49:45.658-0800 I QUERY [conn2] command iibench.$cmd command: insert

      { insert: "purchases_index", ordered: true, documents: 1000 }

      keyUpdates:0 reslen:40 12907ms
      2015-01-04T21:49:53.033-0800 I WRITE [conn9] insert iibench.purchases_index query:

      { _id: ObjectId('54aa25fce4b0fe6bcefcf0d2'), dateandtime: 1420436988731, cashregisterid: 819, customerid: 77578, productid: 8110, price: 780.2229508580186, cf1: "qzafxxbyqkjakijsqouvvzrzhxathevjazprnmeofdznmupmifarnmmsipsbrbjjkokfidkivxtxmwvljcxzbgnovrkuxoddsddqocnvfooeugujzfopvcytzkouesykevvfygrpqteftjrzvkgojy..." }

      ninserted:0 keyUpdates:0 exception: WriteConflict code:112 4204ms

      This also occurs when I use a small cache for WT which should make checkpoint stalls less of an issue.

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            mdcallag Mark Callaghan
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: