ISSUE DESCRIPTION AND IMPACT
This ticket resolves a race condition that may result in logical inconsistency between indexes and collection data. In particular, the following must be true for this issue to race:
- A replica set needs to be experiencing variable replication lag that engages the cache overflow mechanism.
- A workload that would be deleting all the content in a page as well as writing data back into the deleted page area when a checkpoint occurs.
- A workload that inserts more keys in the key range in the section of the WiredTiger tree where there was a large number of previously deleted keys.
The following errors may be symptomatic of this issue:
2019-05-25T05:35:20.781+0000 F REPL [initandlisten] Caught exception during replication recovery: DuplicateKey: E11000 duplicate key error collection: test.test index: _id_ dup key: { : ObjectId('5ce8b8fcff80e8b234715266') }
2019-05-25T05:47:31.52+0000 F STORAGE [conn7299] Unique index cursor seeing multiple records for key { : 1.0 } in index a_1 (table:index-0-4715414094573487683) belonging to collection test.test
2019-04-29T13:45:37.340+0000 F - [conn7299] Fatal Assertion 28608 at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp 1232
DIAGNOSIS AND AFFECTED VERSIONS
This issue affects MongoDB 3.6.0 through 3.6.13 and 4.0.0 through 4.0.9. The validate command may be used to confirm whether a collection has been impacted by this issue.
REMEDIATION AND FIX VERSIONS
A fix for this issue is included in MongoDB versions 3.6.14 and 4.0.10. All users are encouraged to upgrade to take advantage of this fix. To resolve the logical inconsistency, users may perform an initial sync of any affected nodes.
Original Description
When a page is evicted and all keys at the version being reconciled refer to deleted values, but there are other values that need to be kept alive to satisfy transaction visibility rules WiredTiger will keep a reference in the parent page in memory in the WT_REF_DELETED state and that reference will have a pointer into the lookaside file - which is where the versions will be kept available.
If there is a split in the tree - which happens when more content is added to the section of the tree belonging to the internal page - that split operation can free such a WT_REF_DELETED structure even though it is holding a reference to live versions of data. We need to fix the page split code so that it never frees WT_REF_DELETED references with live versions of data.
This issue was discovered while making enhancements to the test/checkpoint program that are described in WT-4750.
- causes
-
WT-4794 Mark lookaside history resolved in all paths
-
- Closed
-