@michaelcahill, I've stared at this one, but I don't see it.
One thread is looking at page->type:
733 parent_decr += size; 734 WT_TRET(__wt_session_fotxn_add(session, pindex, size)); 735 switch (parent->type) { 736 case WT_PAGE_ROW_INT: 737 case WT_PAGE_ROW_LEAF: 738 if ((ikey = __wt_ref_key_instantiated(ref)) == NULL) 739 break; 740 size = sizeof(WT_IKEY) + ikey->size; 741 parent_decr += size; 742 WT_TRET(__wt_session_fotxn_add(session, ikey, size)); 743 break; 744 }
and the page has been evicted/freed.
I want to blame the code that loops until we get a matching ref->home and parent, but I don't see any problems there.
We looked at page->type 20 lines before, line 713, so this must have happened simultaneously, so somehow our parent got evicted while we were splitting into it.
Is there some way an internal page could be selected for eviction after a split but before the child pages appear on it? I don't see how that's possible, either.
================================================================= ==105189==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000bc18f0 at pc 0x4febbf bp 0x7f64b0cdbc50 sp 0x7f64b0cdbc48 READ of size 1 at 0x607000bc18f0 thread T79 #0 0x4febbe in __wt_split_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_split.c>:735 WT-1 0x4fb3e3 in __rec_page_dirty_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:172 WT-2 0x4fb3e3 in __wt_rec_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:88 WT-3 0x4d1bdd in __wt_evict_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:393 WT-4 0x4d1bdd in __wt_evict_lru_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:977 WT-5 0x599585 in __wt_cache_full_check <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cache.i>:93 WT-6 0x599585 in __cursor_enter <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:57 WT-7 0x599585 in __curfile_enter <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:94 WT-8 0x599585 in __cursor_func_init <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:141 WT-9 0x59cc17 in __wt_btcur_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:522:38 WT-10 0x547062 in __curfile_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/cursor/cur_file.c>:262 WT-11 0x452fd7 in row_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:701 WT-12 0x452fd7 in ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:372 WT-13 0x4415b3 in __asan::AsanThread::ThreadStart(unsigned long) (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x4415b3)> WT-14 0x339e807c52 in start_thread (/lib64/libpthread.so.0+0x339e807c52) WT-15 0x339e0f5dbc in __clone (/lib64/libc.so.6+0x339e0f5dbc) 0x607000bc18f0 is located 64 bytes inside of 72-byte region [0x607000bc18b0,0x607000bc18f8) freed by thread T66 here: #0 0x43af54 in free (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x43af54)> WT-1 0x5a50f3 in __wt_page_out <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_discard.c>:97 WT-2 0x4fb738 in __rec_page_dirty_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:192 WT-3 0x4fb738 in __wt_rec_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:88 WT-4 0x4d1bdd in __wt_evict_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:393 WT-5 0x4d1bdd in __wt_evict_lru_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:977 previously allocated by thread T78 here: #0 0x43b0a3 in calloc (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x43b0a3)> WT-1 0x4889cc in __wt_calloc <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/os_posix/os_alloc.c>:38 WT-2 0x4db993 in __wt_page_inmem <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_page.c>:294 WT-3 0x4debe6 in __wt_cache_read <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_read.c>:109 WT-4 0x4da997 in __wt_page_in_func <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_page.c>:46 WT-5 0x52c51e in __wt_page_swap_func <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/btree.i>:704 WT-6 0x52c51e in __wt_row_search <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/row_srch.c>:278 WT-7 0x59b713 in __cursor_row_search <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:158 WT-8 0x59b713 in __wt_btcur_insert <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:393 WT-9 0x5464c7 in __curfile_insert <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/cursor/cur_file.c>:223 Thread T79 created by T0 here: #0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)> WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94 WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180 Thread T66 created by T0 here: #0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)> WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94 WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180 Thread T78 created by T0 here: #0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)> WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94 WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180 SUMMARY: AddressSanitizer: heap-use-after-free <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_split.c>:735 __wt_split_evict Shadow bytes around the buggy address: 0x0c0e801702c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e801702d0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd 0x0c0e801702e0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd 0x0c0e801702f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa 0x0c0e80170300: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd =>0x0c0e80170310: fd fa fa fa fa fa fd fd fd fd fd fd fd fd[fd]fa 0x0c0e80170320: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e80170330: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e80170340: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd 0x0c0e80170350: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd 0x0c0e80170360: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 ASan internal: fe ==105189==ABORTING + status=1 + '[' -f RUNDIR/run ']' + cat RUNDIR/run ############################################ # RUN PARAMETERS ############################################ auto_throttle=1 firstfit=1 # bitcnt not applicable to this run bloom=1 bloom_bit_count=48 bloom_hash_count=7 bloom_oldest=0 cache=3 checkpoints=1 checksum=off chunk_size=7 compaction=0 compression=none data_extend=0 data_source=file delete_pct=12 dictionary=0 file_type=row-store hot_backups=0 huffman_key=0 huffman_value=0 insert_pct=18 internal_key_truncation=1 internal_page_max=14 key_gap=1 key_max=72 key_min=25 leaf_page_max=10 merge_max=19 merge_threads=1 mmap=1 ops=100000 prefix_compression=0 prefix_compression_min=8 repeat_data_pct=7 reverse=0 rows=100000 runs=1 split_pct=58 statistics=0 threads=30 value_max=814 value_min=19 # wiredtiger_config not applicable to this run write_pct=59 ############################################
- related to
-
WT-1 placeholder WT-1
- Closed
-
WT-2 What does metadata look like?
- Closed
-
WT-3 What file formats are required?
- Closed
-
WT-4 Flexible cursor traversals
- Closed
-
WT-5 How does pget work: is it necessary?
- Closed
-
WT-6 Complex schema example
- Closed
-
WT-7 Do we need the handle->err/errx methods?
- Closed
-
WT-8 Do we need table load, bulk-load and/or dump methods?
- Closed
-
WT-9 Does adding schema need to be transactional?
- Closed
-
WT-10 Basic "getting started" tutorial
- Closed
-
WT-11 placeholder #11
- Closed
-
WT-12 Write more examples
- Closed
-
WT-13 Define supported platforms
- Closed
-
WT-14 Windows build
- Closed
-
WT-15 Automated build/test infrastructure
- Closed