Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9477

Don't allow checkpoint triggering internal page split which leads to corrupted internal page in the checkpoint

    • 3
    • Storage Engines - 2022-06-27
    • v6.0

      We have seen a bug in WiredTiger where when restarting from a checkpoint, we notice that the checkpoint is corrupted via a key out-of-order error from MongoDB validation. Upon inspection of the corrupted database, it can be seen that two internal pages are referencing overlapping leaf pages.

      The error message is:

      {"t":{"$date":"2022-05-20T16:42:05.149+00:00"},"s":"I",  "c":"COMMAND",  "id":20514,   "ctx":"conn2","msg":"CMD: validate","attr":{"namespace":"local.oplog.rs","background":false,"full":false,"enforceFastCount":false,"repair":false}}
      {"t":{"$date":"2022-05-20T16:42:05.149+00:00"},"s":"I",  "c":"INDEX",    "id":20303,   "ctx":"conn2","msg":"validating collection","attr":{"namespace":"local.oplog.rs","uuid":{"uuid":{"$uuid":"957ce2cb-8d6b-4f40-b637-1a3f2f3d5ec6"}}}}
      {"t":{"$date":"2022-05-20T16:42:05.449+00:00"},"s":"E",  "c":"STORAGE",  "id":22406,   "ctx":"conn2","msg":"WTCursor::next -- next was not greater than last which is a bug","attr":{"next":"7099859378722836930","last":"7099859378722837016"}}
      {"t":{"$date":"2022-05-20T16:42:05.449+00:00"},"s":"F",  "c":"ASSERT",   "id":23081,   "ctx":"conn2","msg":"Invariant failure","attr":{"expr":"!TestingProctor::instance().isEnabled()","msg":"next was not greater than last","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":2231}}
      {"t":{"$date":"2022-05-20T16:42:05.449+00:00"},"s":"F",  "c":"ASSERT",   "id":23082,   "ctx":"conn2","msg":"\n\n***aborting after invariant() failure\n\n"}
      {"t":{"$date":"2022-05-20T16:42:05.449+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn2","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
      

      Here's an example of the tree structure seen in the checkpoint:

            Assignee:
            haribabu.kommi@mongodb.com Haribabu Kommi
            Reporter:
            chenhao.qu@mongodb.com Chenhao Qu
            Chenhao Qu, Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: