Loading...

Type: Bug
Resolution: Fixed
Priority: Major - P3
Fix Version/s: 4.1.8
Affects Version/s: None
Component/s: Replication
Labels:
- prepare_basic

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Sprint:
Repl 2018-12-17, Repl 2019-01-14, Storage NYC 2019-01-28
Case:
Linked BF Score:
77

To prepare a transaction, a slot is reserved in the oplog, and then the slot's opTime timestamp is used to prepare the transaction in WiredTiger. If a new WiredTiger transaction begins during this window (i.e. after reserving a time, before it is given to WT) and starts reading at a timestamp >= the reserved prepare timestamp, WiredTiger will fail to prepare the transaction because "the prepare timestamp must be later/greater than the latest active read timestamp", failing this invariant.

Example crash (from indexed_insert_large_noindex.js failure in this evergreen patch):

[ShardedClusterFixture:job0:shard1:primary] 2018-12-06T19:01:36.043+0000 E STORAGE  [conn13085] WiredTiger error (22) [1544122896:43869][47134:0x7f8d88774700], WT_SESSION.prepare_transaction: __wt_txn_parse_prepare_timestamp, 722: prepare timestamp 5C09721000000006 not later than an active read timestamp 5c09721000000007 : Invalid argument Raw: [1544122896:43869][47134:0x7f8d88774700], WT_SESSION.prepare_transaction: __wt_txn_parse_prepare_timestamp, 722: prepare timestamp 5C09721000000006 not later than an active read timestamp 5c09721000000007 : Invalid argument
[ShardedClusterFixture:job0:shard1:primary] 2018-12-06T19:01:36.043+0000 F -        [conn13085] Invariant failure: s->prepare_transaction(s, conf.c_str()) resulted in status BadValue: 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 185
[ShardedClusterFixture:job0:shard1:primary] 2018-12-06T19:01:36.044+0000 F -        [conn13085]
[ShardedClusterFixture:job0:shard1:primary] 
[ShardedClusterFixture:job0:shard1:primary] ***aborting after invariant() failure
[ShardedClusterFixture:job0:shard1:primary] 
[ShardedClusterFixture:job0:shard1:primary] 
[ShardedClusterFixture:job0:shard1:primary] 2018-12-06T19:01:36.075+0000 F -        [conn13085] Got signal: 6 (Aborted).
[ShardedClusterFixture:job0:shard1:primary]  0x7f8dbfb638e1 0x7f8dbfb62af9 0x7f8dbfb62fdd 0x7f8dbc1307e0 0x7f8dbbdbf495 0x7f8dbbdc0c75 0x7f8dbe0d8f79 0x7f8dbe1a31ee 0x7f8dbfa9f6c1 0x7f8dbf07e573 0x7f8dbe394c45 0x7f8dbe395595 0x7f8dbe56e834 0x7f8dbe570ca0 0x7f8dbe5728be 0x7f8dbe57369f 0x7f8dbe56003a 0x7f8dbe56c6be 0x7f8dbe56822f 0x7f8dbe56b63d 0x7f8dbf2dbb42 0x7f8dbe565da8 0x7f8dbe569104 0x7f8dbe5673cc 0x7f8dbe5682c1 0x7f8dbe56b63d 0x7f8dbf2dc0b5 0x7f8dbfa9e9c4 0x7f8dbc128aa1 0x7f8dbbe75bdd
[ShardedClusterFixture:job0:shard1:primary] ----- BEGIN BACKTRACE -----
[ShardedClusterFixture:job0:shard1:primary] {"backtrace":[{"b":"7F8DBD672000","o":"24F18E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F8DBD672000","o":"24F0AF9"},{"b":"7F8DBD672000","o":"24F0FDD"},{"b":"7F8DBC121000","o":"F7E0"},{"b":"7F8DBBD8D000","o":"32495","s":"gsignal"},{"b":"7F8DBBD8D000","o":"33C75","s":"abort"},{"b":"7F8DBD672000","o":"A66F79","s":"_ZN5mongo24invariantOKFailedWithMsgEPKcRKNS_6StatusERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j"},{"b":"7F8DBD672000","o":"B311EE","s":"_ZN5mongo22WiredTigerRecoveryUnit17prepareUnitOfWorkEv"},{"b":"7F8DBD672000","o":"242D6C1","s":"_ZN5mongo15WriteUnitOfWork7prepareEv"},{"b":"7F8DBD672000","o":"1A0C573","s":"_ZN5mongo22TransactionParticipant18prepareTransactionEPNS_16OperationContextEN5boost8optionalINS_4repl6OpTimeEEE"},{"b":"7F8DBD672000","o":"D22C45"},{"b":"7F8DBD672000","o":"D23595"},{"b":"7F8DBD672000","o":"EFC834"},{"b":"7F8DBD672000","o":"EFECA0"},{"b":"7F8DBD672000","o":"F008BE"},{"b":"7F8DBD672000","o":"F0169F","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"7F8DBD672000","o":"EEE03A","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"7F8DBD672000","o":"EFA6BE","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"7F8DBD672000","o":"EF622F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7F8DBD672000","o":"EF963D"},{"b":"7F8DBD672000","o":"1C69B42","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"7F8DBD672000","o":"EF3DA8","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"7F8DBD672000","o":"EF7104","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"7F8DBD672000","o":"EF53CC","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"7F8DBD672000","o":"EF62C1","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7F8DBD672000","o":"EF963D"},{"b":"7F8DBD672000","o":"1C6A0B5"},{"b":"7F8DBD672000","o":"242C9C4"},{"b":"7F8DBC121000","o":"7AA1"},{"b":"7F8DBBD8D000","o":"E8BDD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.1.6-32-gb18a6b96d0-patch-5c096a752a60ed2d89759d97", "gitVersion" : "b18a6b96d0c28bc98a1e16b7de0d5f104fd3c937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.el6.x86_64", "version" : "#1 SMP Wed Nov 9 08:03:13 EST 2011", "machine" : "x86_64" }, "somap" : [ { "b" : "7F8DBD672000", "elfType" : 3, "buildId" : "57732CF5367F00101F6255B61CA5CD0571C83E54" }, { "b" : "7FFF242FF000", "elfType" : 3, "buildId" : "08F634A1D22DEFF00461D50A7699DACDC97657BF" }, { "b" : "7F8DBD235000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F0BE1166EDCFFB2422B940D601A1BBD89352D80F" }, { "b" : "7F8DBCE50000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "1EDB45C205A844A75EBBB4F0075E705803FFB85B" }, { "b" : "7F8DBCBE4000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D256E285C5E11D9A99EB04CA7651003A8F67B64E" }, { "b" : "7F8DBC9E0000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1F7E85410384392BC51FA7324961719A10125F31" }, { "b" : "7F8DBC7D8000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "FDF3A36FFFE08375456D59DA959EAB2FC30B6186" }, { "b" : "7F8DBC554000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8A852AC42F0B64F0F30C760EBBCFA3FE4A228F12" }, { "b" : "7F8DBC33E000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, { "b" : "7F8DBC121000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "85104ECFE42C606B31C2D0D0D2E5DACD3286A341" }, { "b" : "7F8DBBD8D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E3AACE76351B6A83390CA065E904EB82FBD1EC7" }, { "b" : "7F8DBD44F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1CC2165E019D43F71FDE0A47AF9F4C8EB5E51963" }, { "b" : "7F8DBBB77000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7F8DBB933000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0C249DF4D77989253CCD859956BF50749308A16A" }, { "b" : "7F8DBB64C000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "624C7056B8BBE6BA758DEF557F516FBDBD01E1FD" }, { "b" : "7F8DBB448000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "57F77704A7F1F4E3689D028D3F9ADD4E77486EC9" }, { "b" : "7F8DBB21C000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "C81673692EEF670BC951EE726490F5D1CAB822F4" }, { "b" : "7F8DBB011000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "03B69EEB8998AC9CA7519A27571BAD976BA4C56D" }, { "b" : "7F8DBAE0E000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "b" : "7F8DBABEF000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
[ShardedClusterFixture:job0:shard1:primary]  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f8dbfb638e1]
[ShardedClusterFixture:job0:shard1:primary]  mongod(+0x24F0AF9) [0x7f8dbfb62af9]
[ShardedClusterFixture:job0:shard1:primary]  mongod(+0x24F0FDD) [0x7f8dbfb62fdd]
[ShardedClusterFixture:job0:shard1:primary]  libpthread.so.0(+0xF7E0) [0x7f8dbc1307e0]
[ShardedClusterFixture:job0:shard1:primary]  libc.so.6(gsignal+0x35) [0x7f8dbbdbf495]
[ShardedClusterFixture:job0:shard1:primary]  libc.so.6(abort+0x175) [0x7f8dbbdc0c75]
[ShardedClusterFixture:job0:shard1:primary]  mongod(_ZN5mongo24invariantOKFailedWithMsgEPKcRKNS_6StatusERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j+0x0) [0x7f8dbe0d8f79]
[ShardedClusterFixture:job0:shard1:primary]  mongod(_ZN5mongo22WiredTigerRecoveryUnit17prepareUnitOfWorkEv+0x2DE) [0x7f8dbe1a31ee]
[ShardedClusterFixture:job0:shard1:primary]  mongod(_ZN5mongo15WriteUnitOfWork7prepareEv+0x31) [0x7f8dbfa9f6c1]
[ShardedClusterFixture:job0:shard1:primary]  mongod(_ZN5mongo22TransactionParticipant18prepareTransactionEPNS_16OperationContextEN5boost8optionalINS_4repl6OpTimeEEE+0x143) [0x7f8dbf07e573]
[ShardedClusterFixture:job0:shard1:primary]  mongod(+0xD22C45) [0x7f8dbe394c45]
[ShardedClusterFixture:job0:shard1:primary]  mongod(+0xD23595) [0x7f8dbe395595]

depends on

SERVER-38832 Allow a transaction to be prepared behind the read timestamp of storage transactions with ignore_prepared=true

Closed

SERVER-38906 Multi-document transactions should not perform timestamped read ahead of all-committed

Closed

has to be done before

SERVER-38569 Unblacklist remove_and_bulk_insert.js concurrency workload from sharded txn suites (currently times out on slow variants)

Closed

SERVER-40145 Re-enable test suites that exercise overlapping prepared and read timestamps

Closed

is related to

SERVER-36382 only snapshot, linearizable, and afterClusterTime reads should block on prepare conflicts

Closed

related to

SERVER-35798 Writing an oplog entry for prepare should not push the lastApplied timestamp

Closed

SERVER-57443 [ephemeralForTest] fix oplog visibility for EFT

Closed

(2 related to)

Details

Description

Attachments

Issue Links

Activity

People

Dates