Loading...

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT10.0.0, 4.4.0-rc6, 4.7.0
Affects Version/s: None
Component/s: None
Labels:
None

Story Points:
5
Sprint:
Storage - Ra 2020-05-04, Storage - Ra 2020-05-18

After ~~WT-5977~~ get merged, a lot of replica sets are failing in mongodb tests with the following backtrace.

[BackgroundSync] "BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"7FA789F0E98A","b":"7FA78714B000","o":"2DC398A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.595","s+":"1DA"},{"a":"7FA789F100F9","b":"7FA78714B000","o":"2DC50F9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"7FA789F0D620","b":"7FA78714B000","o":"2DC2620","s":"_ZN5mongo12_GLOBAL__N_111myTerminateEv","s+":"B0"},{"a":"7FA78A0B9DF6","b":"7FA78714B000","o":"2F6EDF6","s":"_ZN10__cxxabiv111__terminateEPFvvE","s+":"6"},{"a":"7FA78A0B9E31","b":"7FA78714B000","o":"2F6EE31","s":"_ZSt9terminatev","s+":"11"},{"a":"7FA787D97ED9","b":"7FA78714B000","o":"C4CED9","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE.cold.1212","s+":"D5"},{"a":"7FA7884C0A52","b":"7FA78714B000","o":"1375A52","s":"_ZN5mongo4repl12RollbackImpl43_runPhaseFromAbortToReconstructPreparedTxnsEPNS_16OperationContextENS0_23RollBackLocalOperations19RollbackCommonPointE","s+":"442"},{"a":"7FA7884C2977","b":"7FA78714B000","o":"1377977","s":"_ZN5mongo4repl12RollbackImpl11runRollbackEPNS_16OperationContextE","s+":"517"},{"a":"7FA78848A6C0","b":"7FA78714B000","o":"133F6C0","s":"_ZN5mongo4repl14BackgroundSync34_runRollbackViaRecoverToCheckpointEPNS_16OperationContextERKNS_11HostAndPortEPNS0_14OplogInterfaceEPNS0_16StorageInterfaceESt8functionIFPNS_12DBClientBaseEvEE","s+":"240"},{"a":"7FA78848B3A1","b":"7FA78714B000","o":"13403A1","s":"_ZN5mongo4repl14BackgroundSync12_runRollbackEPNS_16OperationContextERKNS_6StatusERKNS_11HostAndPortEiPNS0_16StorageInterfaceE","s+":"A91"},{"a":"7FA787DAA0EB","b":"7FA78714B000","o":"C5F0EB","s":"_ZN5mongo4repl14BackgroundSync8_produceEv.cold.1023","s+":"23BF"},{"a":"7FA78848B866","b":"7FA78714B000","o":"1340866","s":"_ZN5mongo4repl14BackgroundSync12_runProducerEv","s+":"E6"},{"a":"7FA78848BB50","b":"7FA78714B000","o":"1340B50","s":"_ZN5mongo4repl14BackgroundSync4_runEv","s+":"C0"},{"a":"7FA78848BC0C","b":"7FA78714B000","o":"1340C0C","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_4repl14BackgroundSync7startupEPNS3_16OperationContextEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"},{"a":"7FA78A0D587F","b":"7FA78714B000","o":"2F8A87F","s":"execute_native_thread_routine","s+":"F"},{"a":"7FA7859E0AA1","b":"7FA7859D9000","o":"7AA1","s":"start_thread","s+":"D1"},{"a":"7FA78572DC4D","b":"7FA785645000","o":"E8C4D","s":"clone","s+":"6D"}],"processInfo":{"mongodbVersion":"4.4.0-rc3-25-gb6670ad-patch-5eaa6c5b3627e0356f60e30e","gitVersion":"b6670adbbf280289cb3cf06edf30fe7290b0b350","compiledModules":[],"uname":{"sysname":"Linux","release":"2.6.32-220.el6.x86_64","version":"#1 SMP Wed Nov 9 08:03:13 EST 2011","machine":"x86_64"},"somap":[{"b":"7FA78714B000","elfType":3,"buildId":"49BA647FE112BE5FC490C3241FC1F2C6C33F4586"}]}}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA789F0E98A","b":"7FA78714B000","o":"2DC398A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.595","s+":"1DA"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA789F100F9","b":"7FA78714B000","o":"2DC50F9","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA789F0D620","b":"7FA78714B000","o":"2DC2620","s":"_ZN5mongo12_GLOBAL__N_111myTerminateEv","s+":"B0"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78A0B9DF6","b":"7FA78714B000","o":"2F6EDF6","s":"_ZN10__cxxabiv111__terminateEPFvvE","s+":"6"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78A0B9E31","b":"7FA78714B000","o":"2F6EE31","s":"_ZSt9terminatev","s+":"11"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA787D97ED9","b":"7FA78714B000","o":"C4CED9","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE.cold.1212","s+":"D5"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA7884C0A52","b":"7FA78714B000","o":"1375A52","s":"_ZN5mongo4repl12RollbackImpl43_runPhaseFromAbortToReconstructPreparedTxnsEPNS_16OperationContextENS0_23RollBackLocalOperations19RollbackCommonPointE","s+":"442"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA7884C2977","b":"7FA78714B000","o":"1377977","s":"_ZN5mongo4repl12RollbackImpl11runRollbackEPNS_16OperationContextE","s+":"517"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78848A6C0","b":"7FA78714B000","o":"133F6C0","s":"_ZN5mongo4repl14BackgroundSync34_runRollbackViaRecoverToCheckpointEPNS_16OperationContextERKNS_11HostAndPortEPNS0_14OplogInterfaceEPNS0_16StorageInterfaceESt8functionIFPNS_12DBClientBaseEvEE","s+":"240"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.366+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78848B3A1","b":"7FA78714B000","o":"13403A1","s":"_ZN5mongo4repl14BackgroundSync12_runRollbackEPNS_16OperationContextERKNS_6StatusERKNS_11HostAndPortEiPNS0_16StorageInterfaceE","s+":"A91"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA787DAA0EB","b":"7FA78714B000","o":"C5F0EB","s":"_ZN5mongo4repl14BackgroundSync8_produceEv.cold.1023","s+":"23BF"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78848B866","b":"7FA78714B000","o":"1340866","s":"_ZN5mongo4repl14BackgroundSync12_runProducerEv","s+":"E6"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78848BB50","b":"7FA78714B000","o":"1340B50","s":"_ZN5mongo4repl14BackgroundSync4_runEv","s+":"C0"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78848BC0C","b":"7FA78714B000","o":"1340C0C","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_4repl14BackgroundSync7startupEPNS3_16OperationContextEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78A0D587F","b":"7FA78714B000","o":"2F8A87F","s":"execute_native_thread_routine","s+":"F"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA7859E0AA1","b":"7FA7859D9000","o":"7AA1","s":"start_thread","s+":"D1"}}
[js_test:rollback_recovery_commit_transaction_before_stable_timestamp] 2020-04-30T06:38:56.367+0000 d21030| 2020-04-30T06:38:56.366+00:00 I  CONTROL  31427   [BackgroundSync] "  Frame: {frame}","attr":{"frame":{"a":"7FA78572DC4D","b":"7FA785645000","o":"E8C4D","s":"clone","s+":"6D"}}

is caused by

WT-5977 WT_SESSION_NO_RECONCILE flag set by history cursor prevents eviction

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates