Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-569

Handle locking issue

    • Type: Icon: Task Task
    • Resolution: Done
    • WT2.0
    • Affects Version/s: None
    • Component/s: None
    • None

      I came across a hang running test/format with LSM. It doesn't seem to be directly related to LSM.

      I see the following stack traces when the application is hung:

      (gdb) thread apply all where
      
      Thread 2 (Thread 0x7ffff77d6700 (LWP 45983)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 ()
          at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:217
      WT-1  0x00000000004238c3 in __wt_cond_wait (session=0x8c46c0, cond=0x8cb320, 
          usecs=100000) at ../src/os_posix/os_mtx.c:75
      WT-2  0x000000000043e055 in __wt_cache_evict_server (arg=0x8c46c0)
          at ../src/btree/bt_evict.c:167
      WT-3  0x000000383c007d15 in start_thread (arg=0x7ffff77d6700)
          at pthread_create.c:308
      WT-4  0x000000383b8f248d in clone ()
          at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
      
      Thread 1 (Thread 0x7ffff7de1740 (LWP 45974)):
      #0  pthread_rwlock_wrlock ()
          at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:85
      WT-1  0x0000000000423eb0 in __wt_writelock (session=0x8c4cf0, 
          rwlock=0x7ffff0410060) at ../src/os_posix/os_mtx.c:239
      WT-2  0x000000000046487c in __wt_conn_btree_close (session=0x8c4cf0, locked=0)
          at ../src/conn/conn_dhandle.c:480
      WT-3  0x0000000000433c4d in __wt_session_discard_btree (session=0x8c4cf0, 
          dhandle_cache=0x0) at ../src/session/session_dhandle.c:323
      WT-4  0x00000000004300a9 in __session_close_cache (session=0x8c4cf0)
          at ../src/session/session_api.c:39
      WT-5  0x0000000000430375 in __session_close (wt_session=0x8c4cf0, config=0x0)
          at ../src/session/session_api.c:82
      WT-6  0x000000000041d3dd in __lsm_tree_close (session=0x8c48d0, lsm_tree=0x8e2440)
          at ../src/lsm/lsm_tree.c:139
      WT-7  0x000000000041d48a in __wt_lsm_tree_close_all (session=0x8c48d0)
          at ../src/lsm/lsm_tree.c:166
      WT-8  0x000000000041b53f in __wt_connection_close (conn=0x8c2420)
          at ../src/conn/conn_open.c:94
      WT-9  0x0000000000417580 in __conn_close (wt_conn=0x8c2420, config=0x0)
          at ../src/conn/conn_api.c:386
      WT-10 0x0000000000415789 in wts_close () at ../../../test/format/wts.c:269
      WT-11 0x0000000000413a25 in main (argc=0, argv=0x7fffffffe370)
          at ../../../test/format/t.c:158
      

      Thread 1 is the interesting one, it's attempting to get a write lock on a dhandle. The dhandle has a reference count of 0, yet we can't get the write lock.

      The dhandle does not refer to the metadata file, and the session isn't the default session.

      I suspect that we're either failing to cleanup on error from a _wt_session_lock_btree call (though I'd expect to see a different error earlier in that case), or we are possibly racing opening a handle, and leaving something in a bad state.

      The config I used to produce this is:

      ############################################
      #  RUN PARAMETERS
      ############################################
      # bitcnt not applicable to this run
      cache=94
      compression=bzip 
      data_extend=0
      data_source=lsm
      delete_pct=14
      dictionary=0
      file_type=row-store
      huffman_key=0
      huffman_value=0
      insert_pct=40
      internal_key_truncation=0
      internal_page_max=14
      key_gap=4
      key_max=102
      key_min=27
      leaf_page_max=21
      ops=382650
      prefix=1
      repeat_data_pct=37
      reverse=0
      rows=600
      runs=0
      split_pct=65
      threads=10
      value_max=2186
      value_min=3
      #wiredtiger_config=lsm_merge=false
      write_pct=5
      ############################################
      

      The most interesting thing is that it's configuring LSM.

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: