Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31383

WiredTiger error:read checksum error

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.9
    • Component/s: WiredTiger
    • Linux

      A customer's server has stopped working after upgrading from 3.4.5 to 3.4.9.
      Attempting to start the mongod process with the configuration file results in the following log:

      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] MongoDB starting : pid=3040 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] db version v3.4.9
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] modules: none
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] build environment:
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
      2017-10-04T17:44:53.801+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
      2017-10-04T17:44:53.824+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2017-10-04T17:44:53.824+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten]
      2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
      2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
      2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-10-04T17:44:53.838+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507139093:838762][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
      2017-10-04T17:44:53.838+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507139093:838813][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
      2017-10-04T17:44:53.838+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1507139093:838822][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-10-04T17:44:53.838+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
      2017-10-04T17:44:53.838+0000 I -        [initandlisten]
      
      ***aborting after fassert() failure
      
      
      2017-10-04T17:44:53.857+0000 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x5638a678e551 0x5638a678d769 0x5638a678dc4d 0x7f9544922390 0x7f954457c428 0x7f954457e02a 0x5638a5a3af93 0x5638a64a1e56 0x5638a5a4522c 0x5638a5a45451 0x5638a5a456a9 0x5638a7093e1f 0x5638a709436b 0x5638a709095d 0x5638a7095537 0x5638a70af12b 0x5638a70e8493 0x5638a7172d6b 0x5638a71732ad 0x5638a717356c 0x5638a70f6b91 0x5638a71690c8 0x5638a71861f0 0x5638a718673f 0x5638a711c5cc 0x5638a71874f2 0x5638a70ec4b7 0x5638a70e4726 0x5638a6486c2f 0x5638a647f4ac 0x5638a63727a0 0x5638a5a271be 0x5638a5a46a56 0x7f9544567830 0x5638a5aa5a59
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"5638A5220000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5638A5220000","o":"156D769"},{"b":"5638A5220000","o":"156DC4D"},{"b":"7F9544911000","o":"11390"},{"b":"7F9544547000","o":"35428","s":"gsignal"},{"b":"7F9544547000","o":"3702A","s":"abort"},{"b":"5638A5220000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5638A5220000","o":"1281E56"},{"b":"5638A5220000","o":"82522C","s":"__wt_eventv"},{"b":"5638A5220000","o":"825451","s":"__wt_err"},{"b":"5638A5220000","o":"8256A9","s":"__wt_panic"},{"b":"5638A5220000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"5638A5220000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"5638A5220000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"5638A5220000","o":"1E75537"},{"b":"5638A5220000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"5638A5220000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"5638A5220000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"5638A5220000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"5638A5220000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"5638A5220000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"5638A5220000","o":"1F490C8"},{"b":"5638A5220000","o":"1F661F0"},{"b":"5638A5220000","o":"1F6673F"},{"b":"5638A5220000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"5638A5220000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"5638A5220000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"5638A5220000","o":"1EC4726","s":"wiredtiger_open"},{"b":"5638A5220000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5638A5220000","o":"125F4AC"},{"b":"5638A5220000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5638A5220000","o":"8071BE"},{"b":"5638A5220000","o":"826A56","s":"main"},{"b":"7F9544547000","o":"20830","s":"__libc_start_main"},{"b":"5638A5220000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5638A5220000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFEAEF62000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F954589D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9545459000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9545251000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F954504D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9544D44000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9544B2E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9544911000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9544547000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9545B06000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5638a678e551]
      mongod(+0x156D769) [0x5638a678d769]
       mongod(+0x156DC4D) [0x5638a678dc4d]
       libpthread.so.0(+0x11390) [0x7f9544922390]
       libc.so.6(gsignal+0x38) [0x7f954457c428]
       libc.so.6(abort+0x16A) [0x7f954457e02a]
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5638a5a3af93]
       mongod(+0x1281E56) [0x5638a64a1e56]
       mongod(__wt_eventv+0x3D7) [0x5638a5a4522c]
       mongod(__wt_err+0x9D) [0x5638a5a45451]
       mongod(__wt_panic+0x24) [0x5638a5a456a9]
       mongod(__wt_block_extlist_read+0x8F) [0x5638a7093e1f]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x5638a709436b]
       mongod(__wt_block_checkpoint_load+0x26D) [0x5638a709095d]
       mongod(+0x1E75537) [0x5638a7095537]
       mongod(__wt_btree_open+0xB3B) [0x5638a70af12b]
       mongod(__wt_conn_btree_open+0x163) [0x5638a70e8493]
       mongod(__wt_session_get_btree+0xFB) [0x5638a7172d6b]
       mongod(__wt_session_get_btree+0x63D) [0x5638a71732ad]
       mongod(__wt_session_get_btree_ckpt+0x14C) [0x5638a717356c]
       mongod(__wt_curfile_open+0x161) [0x5638a70f6b91]
       mongod(+0x1F490C8) [0x5638a71690c8]
       mongod(+0x1F661F0) [0x5638a71861f0]
       mongod(+0x1F6673F) [0x5638a718673f]
       mongod(__wt_log_scan+0xC7C) [0x5638a711c5cc]
       mongod(__wt_txn_recover+0x572) [0x5638a71874f2]
       mongod(__wt_connection_workers+0x37) [0x5638a70ec4b7]
       mongod(wiredtiger_open+0x1666) [0x5638a70e4726]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5638a6486c2f]
       mongod(+0x125F4AC) [0x5638a647f4ac]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5638a63727a0]
       mongod(+0x8071BE) [0x5638a5a271be]
       mongod(main+0x966) [0x5638a5a46a56]
       libc.so.6(__libc_start_main+0xF0) [0x7f9544567830]
       mongod(_start+0x29) [0x5638a5aa5a59]
      -----  END BACKTRACE  -----
      

      Files in the directory:

      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-0-1422455344354141630.wt
      -rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:47 collection-0-2199009336346498560.wt
      -rw-r--r--  1 mongodb mongodb       61440 Oct  4 17:44 collection-0--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-10--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       36864 Sep 18 20:29 collection-12--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       36864 Sep 18 21:41 collection-14--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:46 collection-16--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-21--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-2-2199009336346498560.wt
      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-23--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb      757760 Sep 18 21:41 collection-2--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       36864 Sep 12 18:11 collection-25--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       12288 Sep 14 18:30 collection-27--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-29--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-31--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:46 collection-33--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-35--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-37--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-39--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-41--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-43--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       53248 Sep 18 10:28 collection-4--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-45--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-47--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-49--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-51--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       45056 Aug 30 14:46 collection-6--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb 33275269120 Sep 18 21:41 collection-8--3433020917902207860.wt
      drwxr-xr-x  2 mongodb mongodb        4096 Sep 18 21:41 diagnostic.data/
      -rw-r--r--  1 mongodb mongodb 51812122624 Sep 18 21:41 index-0-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb   243556352 Sep 18 21:41 index-1-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:45 index-11--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       32768 Jul  3 20:41 index-1-1422455344354141630.wt
      -rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:47 index-1-2199009336346498560.wt
      -rw-r--r--  1 mongodb mongodb       36864 Sep 18 20:29 index-13--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:50 index-1--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:47 index-15--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Sep  1 19:18 index-17--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb      208896 Sep 18 21:41 index-18--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb      294912 Sep 18 21:41 index-19--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb      126976 Sep 18 21:41 index-20--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb  1536196608 Sep 18 21:41 index-2-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:45 index-22--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 15:43 index-24--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       36864 Sep  1 19:20 index-26--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       12288 Sep 14 18:30 index-28--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Sep  1 07:27 index-30--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb   714182656 Sep 18 21:41 index-3-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 index-3-2199009336346498560.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-32--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb      118784 Sep 18 21:41 index-3--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       16384 Aug 30 15:14 index-34--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Sep  1 19:18 index-36--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:45 index-38--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-40--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb   714121216 Sep 18 21:41 index-4-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb       32768 Jun 27 20:48 index-4-2199009336346498560.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-42--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-44--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-46--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-48--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-50--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb  7794946048 Sep 18 21:41 index-5-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-52--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb       36864 Sep 18 10:28 index-5--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb   243535872 Sep 18 21:41 index-6-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb  1599356928 Sep 18 21:41 index-7-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:50 index-7--3433020917902207860.wt
      -rw-r--r--  1 mongodb mongodb  1590890496 Sep 18 21:41 index-8-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb   707862528 Sep 18 21:41 index-9-1143364533558430943.wt
      -rw-r--r--  1 mongodb mongodb   643907584 Sep 18 21:41 index-9--3433020917902207860.wt
      drwxr-xr-x  2 mongodb mongodb        4096 Oct  4 17:44 journal/
      -rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:46 _mdb_catalog.wt
      -rw-r--r--  1 mongodb mongodb           5 Aug 30 14:46 mongod.lock
      -rw-r--r--  1 mongodb mongodb       45056 Sep 18 21:41 sizeStorer.wt
      -rw-r--r--  1 mongodb mongodb          95 Jun 27 20:06 storage.bson
      -rw-r--r--  1 mongodb mongodb          49 Jun 27 20:06 WiredTiger
      -rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 WiredTigerLAS.wt
      -rw-r--r--  1 mongodb mongodb          21 Jun 27 20:06 WiredTiger.lock
      -rw-r--r--  1 mongodb mongodb        1017 Sep 18 21:41 WiredTiger.turtle
      -rw-r--r--  1 mongodb mongodb      241664 Oct  4 17:44 WiredTiger.wt
      

      Any help would be appreciated to get the database back and running. Thanks!

        1. WiredTiger.wt
          236 kB
        2. WiredTiger.turtle
          1.0 kB
        3. WiredTiger.lock
          0.0 kB
        4. SERVER-31383-repair.tar.gz
          15 kB

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            Aharoni Idan Aharoni [X]
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: