Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-71008

Fatal assertion 39001 BadValue: timestamp_transaction 22

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.2.20
    • Component/s: None
    • ALL

      Hi Team,

       

      There was a issue with our mongodb cluster. We have 4 Node cluster (verison 4.2.20)

      of which the primary got failover to one of the existing secondary server. 

       

      Upon analysing the logs, we found the same issue that we had raised couple of months back as part of https://jira.mongodb.org/browse/SERVER-55894 .  

       

      Below is the error log 

      2022-11-02T08:02:34.615+0000 E  STORAGE  [conn30867] WiredTiger error (22) [1667376154:615441][23003:0x7fc88b557700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 676: commit timestamp (1667376154, 11928) is less than the oldest timestamp (1667376154, 11934): Invalid argument Raw: [1667376154:615441][23003:0x7fc88b557700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 676: commit timestamp (1667376154, 11928) is less than the oldest timestamp (1667376154, 11934): Invalid argument
      2022-11-02T08:02:34.616+0000 F  -        [conn30867] Fatal assertion 39001 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1327
      2022-11-02T08:02:34.616+0000 F  -        [conn30867] \n\n***aborting after fassert() failure\n\n
      2022-11-02T08:02:34.690+0000 F  -        [conn30867] Got signal: 6 (Aborted).
       0x56236a09af21 0x56236a09a45e 0x56236a09a4f6 0x7fc8b5c5d8e0 0x7fc8b58d4ca0 0x7fc8b58d6148 0x562368448c1d 0x56236817470b 0x562368557c16 0x562368d79951 0x562368e12afc 0x562368e14765 0x562368bed0ce 0x5623684d3a3f 0x562368d7c7d8 0x562368c5f92c 0x562368c5fd10 0x562368c603c8 0x562368c529ff 0x562368c5059d 0x5623688ee37c 0x5623688ef325 0x5623688f136c 0x5623688f2132 0x5623688df48c 0x5623688ebf55 0x5623688e9737 0x5623688eafc5 0x56236979e102 0x5623688e6ccd 0x5623688e821d 0x5623688e8b66 0x5623688e96fb 0x5623688eafc5 0x56236979e587 0x562369dfee15 0x562369dfee74 0x7fc8b5c5344b 0x7fc8b598e40f
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"562367708000","o":"2992F21","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562367708000","o":"299245E"},{"b":"562367708000","o":"29924F6"},{"b":"7FC8B5C4C000","o":"118E0"},{"b":"7FC8B58A1000","o":"33CA0","s":"gsignal"},{"b":"7FC8B58A1000","o":"35148","s":"abort"},{"b":"562367708000","o":"D40C1D","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},{"b":"562367708000","o":"A6C70B"},{"b":"562367708000","o":"E4FC16","s":"_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE"},{"b":"562367708000","o":"1671951","s":"_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm"},{"b":"562367708000","o":"170AAFC","s":"_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE"},{"b":"562367708000","o":"170C765","s":"_ZN5mongo4repl12logInsertOpsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_bNS_6Date_tE"},{"b":"562367708000","o":"14E50CE","s":"_ZN5mongo14OpObserverImpl9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b"},{"b":"562367708000","o":"DCBA3F","s":"_ZN5mongo18OpObserverRegistry9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b"},{"b":"562367708000","o":"16747D8","s":"_ZN5mongo14CollectionImpl15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEb"},{"b":"562367708000","o":"155792C"},{"b":"562367708000","o":"1557D10"},{"b":"562367708000","o":"15583C8","s":"_ZN5mongo14performInsertsEPNS_16OperationContextERKNS_9write_ops6InsertEb"},{"b":"562367708000","o":"154A9FF"},{"b":"562367708000","o":"154859D"},{"b":"562367708000","o":"11E637C"},{"b":"562367708000","o":"11E7325"},{"b":"562367708000","o":"11E936C"},{"b":"562367708000","o":"11EA132","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"562367708000","o":"11D748C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"562367708000","o":"11E3F55","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E1737","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E2FC5"},{"b":"562367708000","o":"2096102","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"562367708000","o":"11DECCD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"562367708000","o":"11E021D","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"562367708000","o":"11E0B66","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E16FB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E2FC5"},{"b":"562367708000","o":"2096587"},{"b":"562367708000","o":"26F6E15"},{"b":"562367708000","o":"26F6E74"},{"b":"7FC8B5C4C000","o":"744B"},{"b":"7FC8B58A1000","o":"ED40F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.20", "gitVersion" : "15c0712952c356cb711c13a42cb3bca8617d4ebc", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.275-207.503.amzn2.x86_64", "version" : "#1 SMP Wed Apr 13 20:24:50 UTC 2022", "machine" : "x86_64" }, "somap" : [ { "b" : "562367708000", "elfType" : 3, "buildId" : "158183DD7D05EC87B0DE68A9D3A0278785590553" }, { "b" : "7FFEB0352000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "C43776337825FC5B24C9DA856E02319839982F93" }, { "b" : "7FC8B70A7000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "76E323C50375D93039B96955125C9F165AD3F11A" }, { "b" : "7FC8B6E91000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "E9DEEED0BDCA0711B57858014896D968FE2B5D27" }, { "b" : "7FC8B6A3B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "FDF6260976321A07489C660F761CEBDE4CFD47D7" }, { "b" : "7FC8B67CC000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7BE7054050B1BEEB0669CC3A5658FC79748DAD2B" }, { "b" : "7FC8B65C8000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "F15DE2F853AB538930719067AE37E75D4C3227DF" }, { "b" : "7FC8B63C0000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "AB9E6546305ABA17E6910218A3D959B2C7034938" }, { "b" : "7FC8B6080000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "DF9268F8221BCBF25207D9EB17581207C1F60062" }, { "b" : "7FC8B5E6A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2082B3D4C249A26E0B21AB31DE4D251A07312C6" }, { "b" : "7FC8B5C4C000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "EB2DDD809A3C003790EBCBCB2B6F48DAD8047E54" }, { "b" : "7FC8B58A1000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "77254C111C51B5982752D1614611914631524291" }, { "b" : "7FC8B733E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "32A2AF83D52A1E8C4444E52D5281A31294952343" }, { "b" : "7FC8B567A000", "path" : "/lib64/libnghttp2.so.14", "elfType" : 3, "buildId" : "ABE2627C22FDC7706352089D224CA49759CB74F9" }, { "b" : "7FC8B542B000", "path" : "/lib64/libidn2.so.0", "elfType" : 3, "buildId" : "979481B52FE3E6D03440A27BEEB8B42C9F358E03" }, { "b" : "7FC8B5203000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "DB7387A967A1DB497B643672C4E4DA5A9931D347" }, { "b" : "7FC8B4FB7000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DEE4C16B9C3BBBBBF3D3E3A6947EA2918571BDEF" }, { "b" : "7FC8B4CD3000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "86B0F5A67C11C72D678DB9AF142B17DCF08D9FD0" }, { "b" : "7FC8B4AA2000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A3D5952CD58619354C060E2B651490639A6C198C" }, { "b" : "7FC8B489E000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A404F739EA284ED5BE28E9778857D8B016BBD657" }, { "b" : "7FC8B464C000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "E0B0F4EA58494B2D558C56249972502BF76FAE07" }, { "b" : "7FC8B443D000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "005BA2226678E21EAB03B457F1EFE2003791FCE9" }, { "b" : "7FC8B4228000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89D3CC074FA2CAFE035EC68762FACC3EAC173E19" }, { "b" : "7FC8B3F10000", "path" : "/lib64/libunistring.so.0", "elfType" : 3, "buildId" : "DB84428D5AF12394B1E9DCE0DB7C36118C158C3A" }, { "b" : "7FC8B3D01000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "FC001245DC41D133866F7F7F6BBCFA3E2A375FDC" }, { "b" : "7FC8B3AFD000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "C966A5338211832BA7AF1F8EE75170F2FE2C378E" }, { "b" : "7FC8B38E1000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E0C9F3539BBD29B55B91EC5EF78C446E1F24AE7F" }, { "b" : "7FC8B3682000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "9F9FFDFA6BF1398B70FAAA1103599CC736B1C402" }, { "b" : "7FC8B345C000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "2E3428F7750015CCC22F2460442A977CFF9E2413" }, { "b" : "7FC8B312F000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "E924C26079425FD432BDEFA4E6F1D19AC7A9A3ED" }, { "b" : "7FC8B2F00000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "15C43A56B199288739D2BA1D694173AEA7C6B4A5" }, { "b" : "7FC8B2CFC000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "0D96CF7ED26BAA5BA06AD6234575FA8BAB1B01FC" }, { "b" : "7FC8B2AF7000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "467E73CBB0223C865DE55CE2AD8D3DABD8247F7A" }, { "b" : "7FC8B28BB000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "83866E18EABBF34398185BBAC565A15148259CF1" }, { "b" : "7FC8B2694000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "C17759D57199C1AD0E59C2567D9473E2D0878EFF" }, { "b" : "7FC8B245D000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "AB959E1CA807CAE33B552942BE48AB46E57BC124" }, { "b" : "7FC8B21F9000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "803CA9FD9A024206DB65BA5018968E6A74285C68" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56236a09af21]
       mongod(+0x299245E) [0x56236a09a45e]
       mongod(+0x29924F6) [0x56236a09a4f6]
       libpthread.so.0(+0x118E0) [0x7fc8b5c5d8e0]
       libc.so.6(gsignal+0x110) [0x7fc8b58d4ca0]
       libc.so.6(abort+0x148) [0x7fc8b58d6148]
       mongod(_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj+0x0) [0x562368448c1d]
       mongod(+0xA6C70B) [0x56236817470b]
       mongod(_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE+0x1A6) [0x562368557c16]
       mongod(_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm+0x81) [0x562368d79951]
       mongod(_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE+0x9C) [0x562368e12afc]
       mongod(_ZN5mongo4repl12logInsertOpsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_bNS_6Date_tE+0x8B5) [0x562368e14765]
       mongod(_ZN5mongo14OpObserverImpl9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b+0x10E) [0x562368bed0ce]
       mongod(_ZN5mongo18OpObserverRegistry9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b+0xAF) [0x5623684d3a3f]
       mongod(_ZN5mongo14CollectionImpl15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEb+0x298) [0x562368d7c7d8]
       mongod(+0x155792C) [0x562368c5f92c]
       mongod(+0x1557D10) [0x562368c5fd10]
       mongod(_ZN5mongo14performInsertsEPNS_16OperationContextERKNS_9write_ops6InsertEb+0x5F8) [0x562368c603c8]
       mongod(+0x154A9FF) [0x562368c529ff]
       mongod(+0x154859D) [0x562368c5059d]
       mongod(+0x11E637C) [0x5623688ee37c]
       mongod(+0x11E7325) [0x5623688ef325]
       mongod(+0x11E936C) [0x5623688f136c]
       mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x5C2) [0x5623688f2132]
       mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x5623688df48c]
       mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xE5) [0x5623688ebf55]
       mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x117) [0x5623688e9737]
       mongod(+0x11E2FC5) [0x5623688eafc5]
       mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x56236979e102]
       mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x5623688e6ccd]
       mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x5BD) [0x5623688e821d]
       mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x336) [0x5623688e8b66]
       mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x5623688e96fb]
       mongod(+0x11E2FC5) [0x5623688eafc5]
       mongod(+0x2096587) [0x56236979e587]
       mongod(+0x26F6E15) [0x562369dfee15]
       mongod(+0x26F6E74) [0x562369dfee74]
       libpthread.so.0(+0x744B) [0x7fc8b5c5344b]
       libc.so.6(clone+0x3F) [0x7fc8b598e40f]
      -----  END BACKTRACE  ----- 

            Assignee:
            yuan.fang@mongodb.com Yuan Fang
            Reporter:
            srikanta@jungleegames.com Srikanta Patra
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: