Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-11720

Sweep Block handles for multi-handle (tiered) block managers

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • StorEng - Refinement Pipeline

      For non-tiered storage, there is a 1-1 mapping from Btrees to Block manager to WT_BLOCK structs.  For tiered storage Btrees, there is still a single Block manager (WT_BM), but that struct has an array of WT_BLOCKs that represent referenced objects.  The array is built lazily, but is never shrunk.  A simple sweep thread could mark and sweep WT_BLOCKs that are idle.

      This not only removes unused memory, it will also make searches for needed WT_BLOCKs generally faster, and by closing the WT_BLOCK, we'll free a file descriptor when we're using dir_store.  At the moment, running dir_store, we're limited to how many flush_tiers we can do because we'll use approximately (N_FLUSH_TIER * N_TABLES) file descriptors.  I have a test that on Mac has about 10 tables, and after ~1000 flush tiers, we get EMFILE and some various weird behavior happens.

      This limitation with dir_store was discovered while debugging WT-11512.

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            donald.anderson@mongodb.com Donald Anderson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: