Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5959

Application threads evict uncommitted changes in lower isolation levels

      format.sh: job in /home/chenhaoqu/work/wiredtiger/test/format/RUNDIR.2 killed with signal SIGABRT
      format.sh: there may be a core dump associated with this failure
      format.sh: job in /home/chenhaoqu/work/wiredtiger/test/format/RUNDIR.2 failed
      format.sh: /home/chenhaoqu/work/wiredtiger/test/format/RUNDIR.2 log:
          t: process 21456 running
          =================================================================
          ==21456==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000ff23f4 at pc 0x0000008fa37c bp 0x7f02ac488c30 sp 0x7f02ac488c28
          READ of size 1 at 0x607000ff23f4 thread T33
              #0 0x8fa37b in __wt_txn_commit /home/chenhaoqu/work/wiredtiger/build_posix/../src/txn/txn.c:1034:26
              #1 0x85b572 in __session_commit_transaction /home/chenhaoqu/work/wiredtiger/build_posix/../src/session/session_api.c:1677:15
              #2 0x50d8a5 in commit_transaction /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:440:5
              #3 0x50b451 in ops /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:960:13
              #4 0x7f02c5c006da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
              #5 0x7f02c4f4e88e in clone /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      
          0x607000ff23f4 is located 36 bytes inside of 69-byte region [0x607000ff23d0,0x607000ff2415)
          freed by thread T20 here:
              #0 0x4c31d2 in free /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:124:3
              #1 0x76515a in __wt_free_int /home/chenhaoqu/work/wiredtiger/build_posix/../src/os_common/os_alloc.c:301:5
              #2 0xa6052e in __wt_free_update_list /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/bt_discard.c:446:9
              #3 0xb04f81 in __split_multi_inmem_final /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/bt_split.c:1569:13
              #4 0xb03e81 in __wt_split_rewrite /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/bt_split.c:2242:5
              #5 0x6b4e7d in __evict_page_dirty_update /home/chenhaoqu/work/wiredtiger/build_posix/../src/evict/evict_page.c:394:13
              #6 0x6b05f9 in __wt_evict /home/chenhaoqu/work/wiredtiger/build_posix/../src/evict/evict_page.c:219:9
              #7 0x68f011 in __evict_page /home/chenhaoqu/work/wiredtiger/build_posix/../src/evict/evict_lru.c:2246:5
              #8 0x68bc89 in __wt_cache_eviction_worker /home/chenhaoqu/work/wiredtiger/build_posix/../src/evict/evict_lru.c:2336:23
              #9 0x8fdf3e in __wt_cache_eviction_check /home/chenhaoqu/work/wiredtiger/build_posix/../src/include/cache.i:427:13
              #10 0x8fabf1 in __wt_txn_commit /home/chenhaoqu/work/wiredtiger/build_posix/../src/txn/txn.c:1126:9
              #11 0x85b572 in __session_commit_transaction /home/chenhaoqu/work/wiredtiger/build_posix/../src/session/session_api.c:1677:15
              #12 0x50d8a5 in commit_transaction /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:440:5
              #13 0x50b451 in ops /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:960:13
              #14 0x7f02c5c006da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
      
          previously allocated by thread T33 here:
              #0 0x4c374a in calloc /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:155:3
              #1 0x7637af in __wt_calloc /home/chenhaoqu/work/wiredtiger/build_posix/../src/os_common/os_alloc.c:50:14
              #2 0x58d96f in __wt_update_alloc /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/row_modify.c:273:5
              #3 0x58af36 in __wt_row_modify /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/row_modify.c:102:13
              #4 0xa441a5 in __cursor_row_modify_v /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:405:13
              #5 0xa362ae in __btcur_update /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1255:42
              #6 0xa389e1 in __wt_btcur_update /home/chenhaoqu/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1577:13
              #7 0x631a33 in __curfile_update /home/chenhaoqu/work/wiredtiger/build_posix/../src/cursor/cur_file.c:366:5
              #8 0x5148dc in row_update /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:1493:16
              #9 0x50ace7 in ops /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:881:23
              #10 0x7f02c5c006da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
      
          Thread T33 created by T0 here:
              #0 0x4abe6d in pthread_create /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:210:3
              #1 0x789443 in __wt_thread_create /home/chenhaoqu/work/wiredtiger/build_posix/../src/os_posix/os_thread.c:28:5
              #2 0x505f53 in wts_ops /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:188:9
              #3 0x52408f in main /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/t.c:281:13
              #4 0x7f02c4e4eb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
      
          Thread T20 created by T0 here:
              #0 0x4abe6d in pthread_create /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:210:3
              #1 0x789443 in __wt_thread_create /home/chenhaoqu/work/wiredtiger/build_posix/../src/os_posix/os_thread.c:28:5
              #2 0x505f53 in wts_ops /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/ops.c:188:9
              #3 0x52408f in main /home/chenhaoqu/work/wiredtiger/build_posix/test/format/../../../test/format/t.c:281:13
              #4 0x7f02c4e4eb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
      
          SUMMARY: AddressSanitizer: heap-use-after-free /home/chenhaoqu/work/wiredtiger/build_posix/../src/txn/txn.c:1034:26 in __wt_txn_commit
          Shadow bytes around the buggy address:
            0x0c0e801f6420: fd fa fa fa fa fa fd fd fd fd fd fd fd fd fd fa
            0x0c0e801f6430: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
            0x0c0e801f6440: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
            0x0c0e801f6450: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
            0x0c0e801f6460: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
          =>0x0c0e801f6470: fd fd fd fd fd fa fa fa fa fa fd fd fd fd[fd]fd
            0x0c0e801f6480: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
            0x0c0e801f6490: fd fa fa fa fa fa fd fd fd fd fd fd fd fd fd fa
            0x0c0e801f64a0: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
            0x0c0e801f64b0: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
            0x0c0e801f64c0: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
          Shadow byte legend (one shadow byte represents 8 application bytes):
            Addressable:           00
            Partially addressable: 01 02 03 04 05 06 07
            Heap left redzone:       fa
            Freed heap region:       fd
            Stack left redzone:      f1
            Stack mid redzone:       f2
            Stack right redzone:     f3
            Stack after return:      f5
            Stack use after scope:   f8
            Global redzone:          f9
            Global init order:       f6
            Poisoned by user:        f7
            Container overflow:      fc
            Array cookie:            ac
            Intra object redzone:    bb
            ASan internal:           fe
            Left alloca redzone:     ca
            Right alloca redzone:    cb
            Shadow gap:              cc
          ==21456==ABORTING
      
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007f02c4e6d801 in __GI_abort () at abort.c:79
      #2  0x00000000004e1107 in __sanitizer::Abort() ()
          at /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:157
      #3  0x00000000004dfb51 in __sanitizer::Die() ()
          at /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:59
      #4  0x00000000004c7b39 in ~ScopedInErrorReport ()
          at /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_report.cc:187
      #5  0x00000000004c9323 in ReportGenericError ()
          at /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_report.cc:464
      #6  0x00000000004c99eb in __asan_report_load1 ()
          at /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:118
      #7  0x00000000008fa37c in __wt_txn_commit (session=0x7f02c629c188, cfg=0x7f02ac489660) at ../src/txn/txn.c:1034
      #8  0x000000000085b573 in __session_commit_transaction (wt_session=0x7f02c629c188, config=0x0) at ../src/session/session_api.c:1677
      #9  0x000000000050d8a6 in commit_transaction (tinfo=0x62f000118400, prepared=false) at ../../../test/format/ops.c:440
      #10 0x000000000050b452 in ops (arg=0x62f000118400) at ../../../test/format/ops.c:960
      #11 0x00007f02c5c006db in start_thread (arg=0x7f02ac48a700) at pthread_create.c:463
      #12 0x00007f02c4f4e88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      (gdb) f 6
      #6  0x00000000004c99eb in __asan_report_load1 ()
          at /data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:118
      118	/data/mci/bb51bb2e9ea058cc981d517c2029628d/toolchain-builder/tmp/build-llvm.sh-ieZ/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc: No such file or directory.
      (gdb) f 7
      #7  0x00000000008fa37c in __wt_txn_commit (session=0x7f02c629c188, cfg=0x7f02ac489660) at ../src/txn/txn.c:1034
      1034	                if (upd->type == WT_UPDATE_RESERVE) {
      (gdb) ls
      Undefined command: "ls".  Try "help".
      (gdb) *upd
      Undefined command: "".  Try "help".
      (gdb) p *upd
      $1 = {txnid = 1610612742, durable_ts = 0, start_ts = 0, next = 0x607000f19750, size = 30, type = 3 '\003', prepare_state = 0 '\000', flags = 1 '\001',
        data = 0x607000ff23f7 "0000056042/LMNOPQRSTUVWXYZABCD"}
      (gdb) p session->txn
      $2 = {id = 22142, isolation = WT_ISO_READ_COMMITTED, forced_iso = 0, snap_min = 22284, snap_max = 22287, snapshot = 0x6190001cfc80, snapshot_count = 3, txn_logsync = 0,
        commit_timestamp = 0, durable_timestamp = 0, first_commit_timestamp = 0, prepare_timestamp = 0, read_timestamp = 0, durable_timestampq = {tqe_next = 0x0,
          tqe_prev = 0x0}, read_timestampq = {tqe_next = 0x0, tqe_prev = 0x0}, clear_durable_q = false, clear_read_q = false, mod = 0x6190001d0180, mod_alloc = 1120,
        mod_count = 9, logrec = 0x604000032090, notify = 0x0, ckpt_lsn = {l = {offset = 0, file = 0}, file_offset = 0}, ckpt_nsnapshot = 0, ckpt_snapshot = 0x0,
        full_ckpt = false, operation_timeout_us = 0, rollback_reason = 0x0, flags = 4100}
      (gdb)
      

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            chenhao.qu@mongodb.com Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: