We hit a bug in WT-7569 that WiredTiger doesn't handle an out of order timestamp edge case correctly.

The edge case is that mongodb performed multiple updates on the same key within the same transaction. However, these updates are performed with different timestamps and there is an out of order timestamp within them, e.g.,

U@20 -> U@15 -> U@20 -> U@10

The bug in WiredTiger assumes that there would be no out of order timestamp update within a single transaction. But apparently it is wrong. The WiredTiger team has a fix for this problem but we also suspect there may be something wrong within the code of mongodb.

In WiredTiger, we have a check to ensure the commit timestamp of each update is not smaller than the first commit timestamp.

        /*
         * Compare against the commit timestamp of the current transaction. Return an error if the
         * given timestamp is older than the first commit timestamp.
         */
        if (F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) && commit_ts < txn->first_commit_timestamp)
            WT_RET_MSG(session, EINVAL,
              "commit timestamp %s older than the first commit timestamp %s for this transaction",
              __wt_timestamp_to_string(commit_ts, ts_string[0]),
              __wt_timestamp_to_string(txn->first_commit_timestamp, ts_string[1]));

We did an experiment to tight the check to ensure the commit timestamps are strictly in order and our mongodb patch build reports a lot of failures, indicating mongodb is doing out of order timestamp update quite frequently within a single transaction.

Can we explore tightening timestamp usage within a single transaction?