WT-8290 introduced an API to determine whether an uncommitted transaction has been rolled back due to excessive pinned dirty cache pressure. This has been useful for load shedding purposes.

However, there remains a scenario where a single storage transaction that is individually too large to fit in cache is rolled back. One such transaction is going to be rolled back even if it runs in isolation. The rollback reason introduced in WT-8290 would just indicate that the transaction had to be aborted due to cache pressure, which would imply that it may succeed if retried later, while in reality this transaction is not going to ever succeed because it just doesn't fit in memory. This scenario is exemplified in SERVER-61909, where a single document insert with a large number of secondary index entries is rolled back and retried indefinitely.

It would be useful to distinguish between rolled back transactions that may be retried, and rolled back transactions that should not be retried and should just return an error to the client.

We should consider adding the ability to limit the maximum amount of dirty data generated by a transaction, and to roll it back with a specific reason if the limit is exceeded.