Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-693

wtperf failure: corrupted chunk

    • Type: Icon: Task Task
    • Resolution: Done
    • WT1.6.5
    • Affects Version/s: None
    • Component/s: None

      A new failure in wtperf, in an evict-lsm run.

      Thread 7 (Thread 802007400 (LWP 100086/wtperf)):
      #0  0x000000080142544c in nanosleep () from /lib/libc.so.7
      WT-1  0x0000000801424fe5 in usleep () from /lib/libc.so.7
      WT-2  0x0000000800d15268 in usleep () from /lib/libthr.so.3
      WT-3  0x00000000004041b2 in execute_populate (cfg=0x7fffffffd840)
          at ../../../bench/wtperf/wtperf.c:684
      WT-4  0x00000000004051d2 in main (argc=5, argv=0x7fffffffd9a0)
          at ../../../bench/wtperf/wtperf.c:1028
      
      Thread 6 (Thread 802008800 (LWP 298764/wtperf)):
      #0  0x0000000800d1a87c in pthread_kill () from /lib/libthr.so.3
      WT-1  0x0000000800d14b25 in pthread_getschedparam () from /lib/libthr.so.3
      WT-2  0x0000000800d1cc8d in pthread_cond_signal () from /lib/libthr.so.3
      WT-3  0x0000000000414f12 in __wt_cond_wait (session=0x80204b208, 
          cond=0x80201d060, usecs=100000) at ../src/os_posix/os_mtx.c:75
      WT-4  0x000000000042fec1 in __wt_cache_evict_server (arg=0x80204b208)
          at ../src/btree/bt_evict.c:167
      WT-5  0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3
      
      Thread 5 (Thread 802008c00 (LWP 299123/wtperf)):
      #0  0x0000000800d1a87c in pthread_kill () from /lib/libthr.so.3
      WT-1  0x0000000800d14b25 in pthread_getschedparam () from /lib/libthr.so.3
      WT-2  0x0000000800d1cc8d in pthread_cond_signal () from /lib/libthr.so.3
      WT-3  0x0000000000414f39 in __wt_cond_wait (session=0x80204b410, 
          cond=0x80201d1c0, usecs=0) at ../src/os_posix/os_mtx.c:82
      WT-4  0x0000000000459399 in __log_archive_server (arg=0x80204b410)
          at ../src/conn/conn_log.c:134
      WT-5  0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3
      
      Thread 4 (Thread 802009000 (LWP 300133/wtperf)):
      #0  0x000000080142544c in nanosleep () from /lib/libc.so.7
      WT-1  0x0000000801392728 in sleep () from /lib/libc.so.7
      WT-2  0x0000000800d15358 in sleep () from /lib/libthr.so.3
      WT-3  0x00000000004039fc in stat_worker (arg=0x7fffffffd840)
          at ../../../bench/wtperf/wtperf.c:532
      WT-4  0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3
      WT-5  0x0000000000000000 in ?? ()
      
      Thread 3 (Thread 802009800 (LWP 300160/wtperf)):
      #0  0x0000000800d1a87c in pthread_kill () from /lib/libthr.so.3
      WT-1  0x0000000800d14b25 in pthread_getschedparam () from /lib/libthr.so.3
      WT-2  0x0000000800d1cc8d in pthread_cond_signal () from /lib/libthr.so.3
      WT-3  0x0000000000414f12 in __wt_cond_wait (session=0x80204bc30, 
          cond=0x80201d2a0, usecs=100000) at ../src/os_posix/os_mtx.c:75
      WT-4  0x0000000000410a1e in __wt_lsm_merge_worker (vargs=0x80201e4a0)
          at ../src/lsm/lsm_worker.c:113
      WT-5  0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3
      
      Thread 2 (Thread 802009c00 (LWP 300667/wtperf)):
      #0  0x00000008014405f7 in strlen () from /lib/libc.so.7
      WT-1  0x0000000000424dd1 in __wt_session_get_btree (session=0x80204ba28, 
          uri=0x5000 <Address 0x5000 out of bounds>, checkpoint=0x0, cfg=0x0, 
          flags=0) at ../src/session/session_dhandle.c:274
      WT-2  0x0000000000410f77 in __wt_lsm_checkpoint_worker (arg=0x8023c2600)
          at ../src/lsm/lsm_worker.c:287
      WT-3  0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3
      WT-4  0x0000000000000000 in ?? ()
      Cannot access memory at address 0x7ffffeff8000
      
      Thread 1 (Thread 80200a000 (LWP 300668/wtperf)):
      #0  0x000000080144534d in memset () from /lib/libc.so.7
      WT-1  0x000000000049e1b6 in __wt_row_search (session=0x80204b618, 
          cbt=0x803c33100) at ../src/btree/row_srch.c:134
      WT-2  0x0000000000490c73 in __wt_btcur_insert (cbt=0x803c33100)
          at ../src/btree/bt_cursor.c:314
      WT-3  0x000000000045e36f in __curfile_insert (cursor=0x803c33100)
          at ../src/cursor/cur_file.c:223
      WT-4  0x0000000000472a05 in __clsm_put (session=0x80204b618, clsm=0x803c07300, 
          key=0x803c073b8, value=0x803c073d8, position=0)
          at ../src/lsm/lsm_cursor.c:1028
      WT-5  0x00000000004726df in __clsm_insert (cursor=0x803c07300)
          at ../src/lsm/lsm_cursor.c:1119
      #0  0x00000008014405f7 in strlen () from /lib/libc.so.7
      

      Thread 2 is having problems:

      (gdb) thread 2
      [Switching to thread 2 (Thread 802009c00 (LWP 300667/wtperf))]WT-2  0x0000000000410f77 in __wt_lsm_checkpoint_worker (arg=0x8023c2600)
          at ../src/lsm/lsm_worker.c:287
      287				WT_ERR(__wt_session_get_btree(
      (gdb) where
      #0  0x00000008014405f7 in strlen () from /lib/libc.so.7
      WT-1  0x0000000000424dd1 in __wt_session_get_btree (session=0x80204ba28, 
          uri=0x5000 <Address 0x5000 out of bounds>, checkpoint=0x0, cfg=0x0, 
          flags=0) at ../src/session/session_dhandle.c:274
      WT-2  0x0000000000410f77 in __wt_lsm_checkpoint_worker (arg=0x8023c2600)
          at ../src/lsm/lsm_worker.c:287
      WT-3  0x0000000800d130a4 in pthread_getprio () from /lib/libthr.so.3
      WT-4  0x0000000000000000 in ?? ()
      Cannot access memory at address 0x7ffffeff8000
      (gdb) p *chunk
      $5 = {id = 12288, generation = 0, uri = 0x5000 <Address 0x5000 out of bounds>, 
        bloom_uri = 0x2 <Address 0x2 out of bounds>, count = 34428158304, 
        create_ts = {tv_sec = 0, tv_nsec = 0}, txnid_max = 0, flags = 858796901}
      

      It looks like something stepped all over 2, maybe 3 elements of cookie.chunk_array:

      (gdb) p cookie
      $46 = {chunk_array = 0x803855240, chunk_alloc = 160, nchunks = 17}
      
      
      (gdb) set $i=0
      (gdb) while ($i < cookie.nchunks)
       >p *cookie.chunk_array[$i]
       >set $i=$i+1
       >end
      
      
      $26 = {id = 12, generation = 1, uri = 0x80201d100 "file:test-000012.lsm", 
        bloom_uri = 0x0, count = 157874, create_ts = {tv_sec = 1379613352, 
          tv_nsec = 2911135}, txnid_max = 0, flags = 20}
      $27 = {id = 21, generation = 1, uri = 0x8038c2d80 "file:test-000021.lsm", 
        bloom_uri = 0x80383b960 "file:test-000021.bf", count = 144496, create_ts = {
          tv_sec = 1379613352, tv_nsec = 922935045}, txnid_max = 0, flags = 21}
      
      
      $28 = {id = 12288, generation = 0, 
        uri = 0x5000 <Address 0x5000 out of bounds>, 
        bloom_uri = 0x2 <Address 0x2 out of bounds>, count = 34428158304, 
        create_ts = {tv_sec = 0, tv_nsec = 0}, txnid_max = 0, flags = 858796901}
      $29 = {id = 1664626992, generation = 879046968, 
        uri = 0x3161366434376437 <Address 0x3161366434376437 out of bounds>, 
        bloom_uri = 0x6432346531383538 <Address 0x6432346531383538 out of bounds>, 
        count = 3906979575111247205, create_ts = {tv_sec = 7293077550088401208, 
          tv_nsec = 3474580131273651042}, txnid_max = 3486743108453216312, 
        flags = 858796901}
      $30 = {id = 22, generation = 0, uri = 0x0, bloom_uri = 0x0, count = 15200, 
        create_ts = {tv_sec = 1379613352, tv_nsec = 962527580}, txnid_max = 347034, 
        flags = 28}
      
      
      $31 = {id = 23, generation = 0, uri = 0x8046213c0 "file:test-000023.lsm", 
        bloom_uri = 0x0, count = 11136, create_ts = {tv_sec = 1379613353, 
          tv_nsec = 24931311}, txnid_max = 358191, flags = 24}
      $32 = {id = 24, generation = 0, uri = 0x8047fb760 "file:test-000024.lsm", 
        bloom_uri = 0x0, count = 10643, create_ts = {tv_sec = 1379613353, 
          tv_nsec = 740860947}, txnid_max = 368836, flags = 24}
      $33 = {id = 25, generation = 0, uri = 0x8038a6760 "file:test-000025.lsm", 
        bloom_uri = 0x0, count = 10560, create_ts = {tv_sec = 1379613353, 
          tv_nsec = 770976891}, txnid_max = 379400, flags = 24}
      $34 = {id = 26, generation = 0, uri = 0x8038a6800 "file:test-000026.lsm", 
        bloom_uri = 0x0, count = 11320, create_ts = {tv_sec = 1379613353, 
          tv_nsec = 800130635}, txnid_max = 390726, flags = 24}
      $35 = {id = 27, generation = 0, uri = 0x804621d80 "file:test-000027.lsm", 
        bloom_uri = 0x0, count = 20593, create_ts = {tv_sec = 1379613353, 
          tv_nsec = 858694709}, txnid_max = 411321, flags = 16}
      $36 = {id = 28, generation = 0, uri = 0x804621e20 "file:test-000028.lsm", 
        bloom_uri = 0x0, count = 20595, create_ts = {tv_sec = 1379613353, 
          tv_nsec = 885188012}, txnid_max = 431922, flags = 16}
      $37 = {id = 30, generation = 0, uri = 0x803bfb200 "file:test-000030.lsm", 
        bloom_uri = 0x0, count = 10295, create_ts = {tv_sec = 1379613354, 
          tv_nsec = 74600087}, txnid_max = 442220, flags = 16}
      $38 = {id = 31, generation = 0, uri = 0x803bfb2a0 "file:test-000031.lsm", 
        bloom_uri = 0x0, count = 11101, create_ts = {tv_sec = 1379613354, 
          tv_nsec = 103767216}, txnid_max = 453325, flags = 16}
      $39 = {id = 32, generation = 0, uri = 0x803837fe0 "file:test-000032.lsm", 
        bloom_uri = 0x0, count = 11241, create_ts = {tv_sec = 1379613354, 
          tv_nsec = 160285327}, txnid_max = 464570, flags = 16}
      $40 = {id = 33, generation = 0, uri = 0x803b5a160 "file:test-000033.lsm", 
        bloom_uri = 0x0, count = 11064, create_ts = {tv_sec = 1379613354, 
          tv_nsec = 192039607}, txnid_max = 475638, flags = 16}
      $41 = {id = 34, generation = 0, uri = 0x803b5a200 "file:test-000034.lsm", 
        bloom_uri = 0x0, count = 10391, create_ts = {tv_sec = 1379613354, 
          tv_nsec = 222324073}, txnid_max = 486043, flags = 0}
      $42 = {id = 35, generation = 0, uri = 0x8038a5720 "file:test-000035.lsm", 
        bloom_uri = 0x0, count = 8912, create_ts = {tv_sec = 1379613354, 
          tv_nsec = 291875220}, txnid_max = 494955, flags = 0}
      (gdb) 
      

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: