Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-48259

MongoDB 4.0.17 crashing in Ubuntu 16.04

XMLWordPrintableJSON

    • ALL

      I have installed Mongo container for my product to log data.

      Here is the version of Mongo for your reference.

      MongoDB shell version v4.0.17
      git version: 0bc918c73390f6e4d6349660e4cd233f5900b69a
      OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
      allocator: tcmalloc
      modules: none
      build environment:
      distmod: ubuntu1604
      distarch: x86_64
      target_arch: x86_64

       root@9e31c7f2d640:/# df -h
      Filesystem Size Used Avail Use% Mounted on
      overlay 38G 3.1G 33G 9% /
      tmpfs 64M 0 64M 0% /dev
      tmpfs 2.6G 0 2.6G 0% /sys/fs/cgroup
      shm 64M 0 64M 0% /dev/shm
      /dev/sda1 38G 3.1G 33G 9% /data/db
      tmpfs 2.6G 0 2.6G 0% /proc/asound
      tmpfs 2.6G 0 2.6G 0% /proc/acpi
      tmpfs 2.6G 0 2.6G 0% /proc/scsi
      tmpfs 2.6G 0 2.6G 0% /sys/firmware

      After logging data for a couple of days I am getting data corruption issues. This is resulting in data loss for me.

       

      2020-05-17T06:44:22.673+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
 2020-05-17T06:44:22.681+0000 I CONTROL [initandlisten] MongoDB starting : pid=6 port=27017 dbpath=/data/db 64-bit host=9e31c7f2d640
 2020-05-17T06:44:22.681+0000 I CONTROL [initandlisten] db version v4.0.18
 2020-05-17T06:44:22.681+0000 I CONTROL [initandlisten] git version: 6883bdfb8b8cff32176b1fd176df04da9165fd67
 2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
 2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] allocator: tcmalloc
 2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] modules: none
 2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] build environment:
 2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] distmod: ubuntu1604
 2020-05-17T06:44:22.683+0000 I CONTROL [initandlisten] distarch: x86_64
 2020-05-17T06:44:22.683+0000 I CONTROL [initandlisten] target_arch: x86_64
 2020-05-17T06:44:22.683+0000 I CONTROL [initandlisten] options: {}
 2020-05-17T06:44:22.684+0000 W STORAGE [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
 2020-05-17T06:44:22.684+0000 I STORAGE [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
 2020-05-17T06:44:22.684+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
 2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] 
 2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
 2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] ** See [http://dochub.mongodb.org/core/prodnotes-filesystem]
 2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=2148M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
 2020-05-17T06:44:24.298+0000 E STORAGE [initandlisten] WiredTiger error (-31803) [1589697864:298835][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_turtle_read, 341: WiredTiger.turtle: fatal turtle file read error: WT_NOTFOUND: item not found Raw: [1589697864:298835][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_turtle_read, 341: WiredTiger.turtle: fatal turtle file read error: WT_NOTFOUND: item not found
 2020-05-17T06:44:24.299+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1589697864:299514][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1589697864:299514][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
 2020-05-17T06:44:24.299+0000 F - [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 420
 2020-05-17T06:44:24.299+0000 F - [initandlisten]

***aborting after fassert() failure

2020-05-17T06:44:24.369+0000 F - [initandlisten] Got signal: 6 (Aborted).
 0x55a2f0532031 0x55a2f0531249 0x55a2f053172d 0x7f8a9a32e390 0x7f8a99f88428 0x7f8a99f8a02a 0x55a2eeae8761 0x55a2eebec4d6 0x55a2eec56a99 0x55a2eea6fac0 0x55a2eea6feda 0x55a2eec22bb0 0x55a2eec20bdc 0x55a2eec00580 0x55a2eec5532d 0x55a2eec5593f 0x55a2eec55bcc 0x55a2eecd4e16 0x55a2eec50ca8 0x55a2eec200ce 0x55a2eec201ab 0x55a2eebfee83 0x55a2eebc03fa 0x55a2eebc6594 0x55a2eeba63d8 0x55a2ef2c7452 0x55a2eeb56986 0x55a2eeb5ab45 0x55a2eeaea2f9 0x7f8a99f73830 0x55a2eeb55299
 ----- BEGIN BACKTRACE -----

{"backtrace":[\\{"b":"55A2EE0C0000","o":"2472031","s":"_ZN5mongo15printStackTraceERSo"}

,\{"b":"55A2EE0C0000","o":"2471249"},\{"b":"55A2EE0C0000","o":"247172D"},\{"b":"7F8A9A31D000","o":"11390"},\{"b":"7F8A99F53000","o":"35428","s":"gsignal"},\{"b":"7F8A99F53000","o":"3702A","s":"abort"},{"b":"55A2EE0C0000","o":"A28761","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"55A2EE0C0000","o":"B2C4D6"},\{"b":"55A2EE0C0000","o":"B96A99"},{"b":"55A2EE0C0000","o":"9AFAC0","s":"__wt_err_func"},{"b":"55A2EE0C0000","o":"9AFEDA","s":"__wt_panic"},{"b":"55A2EE0C0000","o":"B62BB0","s":"__wt_turtle_read"},{"b":"55A2EE0C0000","o":"B60BDC","s":"__wt_metadata_search"},{"b":"55A2EE0C0000","o":"B40580","s":"__wt_conn_dhandle_open"},{"b":"55A2EE0C0000","o":"B9532D","s":"__wt_session_get_dhandle"},{"b":"55A2EE0C0000","o":"B9593F","s":"__wt_session_get_dhandle"},{"b":"55A2EE0C0000","o":"B95BCC","s":"__wt_session_get_btree_ckpt"},{"b":"55A2EE0C0000","o":"C14E16","s":"__wt_curfile_open"},\{"b":"55A2EE0C0000","o":"B90CA8"},{"b":"55A2EE0C0000","o":"B600CE","s":"__wt_metadata_cursor_open"},{"b":"55A2EE0C0000","o":"B601AB","s":"__wt_metadata_cursor"},\{"b":"55A2EE0C0000","o":"B3EE83","s":"wiredtiger_open"},{"b":"55A2EE0C0000","o":"B003FA","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_"},{"b":"55A2EE0C0000","o":"B06594","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb"},\{"b":"55A2EE0C0000","o":"AE63D8"},\{"b":"55A2EE0C0000","o":"1207452","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},\{"b":"55A2EE0C0000","o":"A96986"},{"b":"55A2EE0C0000","o":"A9AB45","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},\{"b":"55A2EE0C0000","o":"A2A2F9","s":"main"},\{"b":"7F8A99F53000","o":"20830","s":"__libc_start_main"},\{"b":"55A2EE0C0000","o":"A95299","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.0.18", "gitVersion" : "6883bdfb8b8cff32176b1fd176df04da9165fd67", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "4.14.154-boot2docker", "version" : "#1 SMP Thu Nov 14 19:19:08 UTC 2019", "machine" : "x86_64" }

, "somap" : [ \{ "b" : "55A2EE0C0000", "elfType" : 3, "buildId" : "9BAA54C4A92F5506E39F152D9D3F90BD0DA0526F" }, \{ "b" : "7FFDF51B0000", "elfType" : 3, "buildId" : "ACA410FCB9C0B0653F00BCE6E8284FD17E9006CF" }, \{ "b" : "7F8A9B72E000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "03F16B86D21AA657A636EB6D7977D285E5B111E4" }, \{ "b" : "7F8A9B513000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, \{ "b" : "7F8A9B0CE000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, \{ "b" : "7F8A9AE65000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, \{ "b" : "7F8A9AC61000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, \{ "b" : "7F8A9AA59000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, \{ "b" : "7F8A9A750000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, \{ "b" : "7F8A9A53A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, \{ "b" : "7F8A9A31D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, \{ "b" : "7F8A99F53000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, \{ "b" : "7F8A9B99D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, \{ "b" : "7F8A99D20000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, \{ "b" : "7F8A99B04000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, \{ "b" : "7F8A998BA000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, \{ "b" : "7F8A996AB000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E150CC35D09A20D93B4D936D72AA7637DABC978D" }, \{ "b" : "7F8A9945A000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3D51E97C8DDD5F9EA1674954F8FA8BB2712E64B5" }, \{ "b" : "7F8A99240000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "E2357F88F4E53F0DE76266532DDDCB7FEB6F87A7" }, \{ "b" : "7F8A98F10000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "85FA1D993E14AC403E56E2D45CAE00ACFBD5766C" }, \{ "b" : "7F8A98CDD000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, \{ "b" : "7F8A98AA7000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, \{ "b" : "7F8A98827000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, \{ "b" : "7F8A98555000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, \{ "b" : "7F8A98326000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, \{ "b" : "7F8A98122000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "E6F8CA010177433A1298DAB8912203E5A5EBB9BA" }, \{ "b" : "7F8A97F17000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, \{ "b" : "7F8A97CFC000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "4FE2C629045AB0AF26EC5D7F8A3D64B9F8CB21B2" }, \{ "b" : "7F8A97ABB000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, \{ "b" : "7F8A97857000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, \{ "b" : "7F8A97644000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, \{ "b" : "7F8A97440000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, \{ "b" : "7F8A97237000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, \{ "b" : "7F8A96FAD000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, \{ "b" : "7F8A96D0B000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, \{ "b" : "7F8A96AD8000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, \{ "b" : "7F8A968C2000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, \{ "b" : "7F8A966BA000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, \{ "b" : "7F8A96491000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, \{ "b" : "7F8A96282000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, \{ "b" : "7F8A96037000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, \{ "b" : "7F8A95D62000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "F0615D615CC5E394B0A73D12AA1382AA872DD1F0" }, \{ "b" : "7F8A95B2A000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FD61CA7A6D603E94E5EFD5C88D8810AE104FCF40" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55a2f0532031]
 mongod(+0x2471249) [0x55a2f0531249]
 mongod(+0x247172D) [0x55a2f053172d]
 libpthread.so.0(+0x11390) [0x7f8a9a32e390]
 libc.so.6(gsignal+0x38) [0x7f8a99f88428]
 libc.so.6(abort+0x16A) [0x7f8a99f8a02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55a2eeae8761]
 mongod(+0xB2C4D6) [0x55a2eebec4d6]
 mongod(+0xB96A99) [0x55a2eec56a99]
 mongod(__wt_err_func+0x90) [0x55a2eea6fac0]
 mongod(__wt_panic+0x39) [0x55a2eea6feda]
 mongod(__wt_turtle_read+0x2B0) [0x55a2eec22bb0]
 mongod(__wt_metadata_search+0x1CC) [0x55a2eec20bdc]
 mongod(__wt_conn_dhandle_open+0xA0) [0x55a2eec00580]
 mongod(__wt_session_get_dhandle+0xED) [0x55a2eec5532d]
 mongod(__wt_session_get_dhandle+0x6FF) [0x55a2eec5593f]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55a2eec55bcc]
 mongod(__wt_curfile_open+0x56) [0x55a2eecd4e16]
 mongod(+0xB90CA8) [0x55a2eec50ca8]
 mongod(__wt_metadata_cursor_open+0x6E) [0x55a2eec200ce]
 mongod(__wt_metadata_cursor+0x4B) [0x55a2eec201ab]
 mongod(wiredtiger_open+0x1A13) [0x55a2eebfee83]
 mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0xAA) [0x55a2eebc03fa]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb+0x674) [0x55a2eebc6594]
 mongod(+0xAE63D8) [0x55a2eeba63d8]
 mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x552) [0x55a2ef2c7452]
 mongod(+0xA96986) [0x55a2eeb56986]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0xF95) [0x55a2eeb5ab45]
 mongod(main+0x9) [0x55a2eeaea2f9]
 libc.so.6(__libc_start_main+0xF0) [0x7f8a99f73830]
 mongod(_start+0x29) [0x55a2eeb55299]
 ----- END BACKTRACE -----

       

      I would like to know the reason this might happen. Is there any thing which I am missing in configuration ?

      Attached the complete error log & diagnostic data for reference.

      Thanks

      -Harshad 

       

        1. diag.tar.gz
          20.35 MB
          Harshad Bhagwat
        2. err.rar
          3.66 MB
          Harshad Bhagwat

            Assignee:
            carl.champain@mongodb.com Carl Champain (Inactive)
            Reporter:
            greathary@gmail.com Harshad Bhagwat
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: