Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9830

Do not allow the fast truncate on-disk state to go backwards because of visibility

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • WT11.1.0, 6.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • 3
    • Storage Engines - 2022-10-03
    • v6.1

      In this test, format reports a mirror mismatch and while it's dumping the pages ASAN detects a use-after-free error.

      https://evergreen.mongodb.com/task/wiredtiger_ubuntu2004_stress_tests_race_condition_stress_sanitizer_test_3_2537dbfc7938d44113f355000c677bc8badb2969_22_09_04_04_27_24

      Here's the ASAN report:

       [2022/09/04 08:26:29.427]     ==24574==ERROR: AddressSanitizer: heap-use-after-free on address 0x633000594800 at pc 0x7f8dc626da1e bp 0x7f8db2734a00 sp 0x7f8db27349f8
       [2022/09/04 08:26:29.427]     READ of size 16 at 0x633000594800 thread T80
       [2022/09/04 08:26:29.427]         #0 0x7f8dc626da1d in __wt_lex_compare_skip /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/include/btree_cmp_inline.h:232:21
       [2022/09/04 08:26:29.427]         #1 0x7f8dc626b768 in __wt_row_search /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/btree/row_srch.c:375:23
       [2022/09/04 08:26:29.427]         #2 0x7f8dc619cad4 in __cursor_row_search /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/btree/bt_cursor.c:501:5
       [2022/09/04 08:26:29.427]         #3 0x7f8dc619f780 in __wt_btcur_search_near /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/btree/bt_cursor.c:967:13
       [2022/09/04 08:26:29.427]         #4 0x7f8dc62fe604 in __curfile_search_near /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/cursor/cur_file.c:348:5
       [2022/09/04 08:26:29.427]         #5 0x4f94be in table_dump_page /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/util.c:320:11
       [2022/09/04 08:26:29.427]         #6 0x4fb5ce in table_verify_mirror /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/verify.c:297:29
       [2022/09/04 08:26:29.427]         #7 0x4fa2d1 in wts_verify /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/verify.c:365:13
       [2022/09/04 08:26:29.427]         #8 0x4cd901 in check_copy /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/backup.c:70:5
       [2022/09/04 08:26:29.427]         #9 0x4ca224 in backup /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/backup.c:651:13
       [2022/09/04 08:26:29.427]         #10 0x7f8dc5dc2608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
       [2022/09/04 08:26:29.427]         #11 0x7f8dc5b6b132 in clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       [2022/09/04 08:26:29.427]     0x633000594800 is located 0 bytes inside of 102400-byte region [0x633000594800,0x6330005ad800)
       [2022/09/04 08:26:29.427]     freed by thread T80 here:
       [2022/09/04 08:26:29.427]         #0 0x498262 in free /data/mci/3c3c046b1b46b72eb7f046666a5afd22/toolchain-builder/tmp/build-llvm-v4.sh-DSy/llvm-project-llvmorg/compiler-rt/lib/asan/asan_malloc_linux.cpp:127:3
       [2022/09/04 08:26:29.427]         #1 0x4dcfc5 in key_gen_teardown /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/kv.c:135:5
       [2022/09/04 08:26:29.427]         #2 0x4f9482 in table_dump_page /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/util.c:317:9
       [2022/09/04 08:26:29.427]         #3 0x4fb5ce in table_verify_mirror /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/verify.c:297:29
       [2022/09/04 08:26:29.427]         #4 0x4fa2d1 in wts_verify /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/verify.c:365:13
       [2022/09/04 08:26:29.427]         #5 0x4cd901 in check_copy /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/backup.c:70:5
       [2022/09/04 08:26:29.427]         #6 0x4ca224 in backup /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/backup.c:651:13
       [2022/09/04 08:26:29.427]         #7 0x7f8dc5dc2608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
       [2022/09/04 08:26:29.427]     previously allocated by thread T80 here:
       [2022/09/04 08:26:29.427]         #0 0x4984cd in malloc /data/mci/3c3c046b1b46b72eb7f046666a5afd22/toolchain-builder/tmp/build-llvm-v4.sh-DSy/llvm-project-llvmorg/compiler-rt/lib/asan/asan_malloc_linux.cpp:145:3
       [2022/09/04 08:26:29.427]         #1 0x5062ce in dmalloc /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/utility/misc.c:397:14
       [2022/09/04 08:26:29.427]         #2 0x4dcd0f in key_gen_init /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/kv.c:118:9
       [2022/09/04 08:26:29.427]         #3 0x4f9432 in table_dump_page /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/util.c:314:9
       [2022/09/04 08:26:29.427]         #4 0x4fb5ce in table_verify_mirror /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/verify.c:297:29
       [2022/09/04 08:26:29.427]         #5 0x4fa2d1 in wts_verify /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/verify.c:365:13
       [2022/09/04 08:26:29.427]         #6 0x4cd901 in check_copy /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/backup.c:70:5
       [2022/09/04 08:26:29.427]         #7 0x4ca224 in backup /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/backup.c:651:13
       [2022/09/04 08:26:29.427]         #8 0x7f8dc5dc2608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
       [2022/09/04 08:26:29.427]     Thread T80 created by T0 here:
       [2022/09/04 08:26:29.427]         #0 0x482bcc in pthread_create /data/mci/3c3c046b1b46b72eb7f046666a5afd22/toolchain-builder/tmp/build-llvm-v4.sh-DSy/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:205:3
       [2022/09/04 08:26:29.427]         #1 0x7f8dc645acff in __wt_thread_create /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/os_posix/os_thread.c:28:5
       [2022/09/04 08:26:29.427]         #2 0x4deb65 in operations /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/ops.c:303:9
       [2022/09/04 08:26:29.427]         #3 0x4f48d1 in main /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../test/format/t.c:376:9
       [2022/09/04 08:26:29.427]         #4 0x7f8dc5a70082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
       [2022/09/04 08:26:29.427]     SUMMARY: AddressSanitizer: heap-use-after-free /data/mci/eafdffe0c4388990d054e32cc91b55c6/wiredtiger/cmake_build/../src/include/btree_cmp_inline.h:232:21 in __wt_lex_compare_skip
       [2022/09/04 08:26:29.427]     Shadow bytes around the buggy address:
       [2022/09/04 08:26:29.427]       0x0c66800aa8b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
       [2022/09/04 08:26:29.427]       0x0c66800aa8c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
       [2022/09/04 08:26:29.427]       0x0c66800aa8d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
       [2022/09/04 08:26:29.427]       0x0c66800aa8e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
       [2022/09/04 08:26:29.427]       0x0c66800aa8f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
       [2022/09/04 08:26:29.427]     =>0x0c66800aa900:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
       [2022/09/04 08:26:29.427]       0x0c66800aa910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
       [2022/09/04 08:26:29.427]       0x0c66800aa920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
       [2022/09/04 08:26:29.427]       0x0c66800aa930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
       [2022/09/04 08:26:29.427]       0x0c66800aa940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
       [2022/09/04 08:26:29.427]       0x0c66800aa950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
       [2022/09/04 08:26:29.427]     Shadow byte legend (one shadow byte represents 8 application bytes):
       [2022/09/04 08:26:29.427]       Addressable:           00
       [2022/09/04 08:26:29.427]       Partially addressable: 01 02 03 04 05 06 07
       [2022/09/04 08:26:29.427]       Heap left redzone:       fa
       [2022/09/04 08:26:29.427]       Freed heap region:       fd
       [2022/09/04 08:26:29.427]       Stack left redzone:      f1
       [2022/09/04 08:26:29.427]       Stack mid redzone:       f2
       [2022/09/04 08:26:29.427]       Stack right redzone:     f3
       [2022/09/04 08:26:29.427]       Stack after return:      f5
       [2022/09/04 08:26:29.427]       Stack use after scope:   f8
       [2022/09/04 08:26:29.428]       Global redzone:          f9
       [2022/09/04 08:26:29.428]       Global init order:       f6
       [2022/09/04 08:26:29.428]       Poisoned by user:        f7
       [2022/09/04 08:26:29.428]       Container overflow:      fc
       [2022/09/04 08:26:29.428]       Array cookie:            ac
       [2022/09/04 08:26:29.428]       Intra object redzone:    bb
       [2022/09/04 08:26:29.428]       ASan internal:           fe
       [2022/09/04 08:26:29.428]       Left alloca redzone:     ca
       [2022/09/04 08:26:29.428]       Right alloca redzone:    cb
       [2022/09/04 08:26:29.428]       Shadow gap:              cc

      The same thread (T80) allocated memory in key_gen_init() and then freed it in key_gen_teardown() and subsequently accessed the same memory in __wt_row_search()

       

            Assignee:
            haribabu.kommi@mongodb.com Haribabu Kommi
            Reporter:
            keith.smith@mongodb.com Keith Smith
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: