Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38588

Hybrid index builds do not work when applied concurrently with prepared transactions on secondaries

    • Fully Compatible
    • Storage NYC 2019-03-11
    • 22

      Original text:

      Test setup: 

      • Multi shard 
      • collection is sharded with _id: hashed 
         
        The evergreen validateCollection hook fails semi-frequently after running the background_index_multikey.js test. Based on initial investigation, it looks like one of the indexes in the secondary is missing a key entry. The primary appears to have passed the validation. Note: test is currently blacklisted.

      Because transactions yield their locks on secondaries (SERVER-37199), a concurrent hybrid background index build can conflict in a way that leads to lost writes into building indexes (i.e. corruption) on secondaries.

      In this example, a background index build on {a: 1} is concurrent with an insert of a document {a: 0} in a transaction while applied on a secondary.

      • The background index build converts its X lock to an IX lock while collection scanning. It creates a temporary side-writes table to accept all index key insertions during the build.
      • A document {a: 0}is inserted in a transaction and prepared. The key for a: 0 is inserted into the side-writes table as part of the same transaction. When applied on a secondary, it drops its IX locks.
      • The background index build takes an X lock, uncontested, and drains the side-writes table. Because the insert into the side-writes table was part of a prepared, but uncommitted transaction, it is invisible to the index builder. The table is then dropped on completion.
      • The transaction is finally committed, but its side-write is committed to a now-deleted table. The inserted key is now lost forever and the resulting index is corrupted.

      On a primary, our locks prevent this from happening, but because an index build can complete while a prepared transaction is active, we can lose writes into building indexes.

      edit: louis.williams

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            randolph@mongodb.com Randolph Tan
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: