Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-11173

Reduce the default dirty cache limit with large caches

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Engines
    • StorEng - Refinement Pipeline

      At the moment, the default value for eviction_dirty_trigger is 20% of the configured cache size. That makes sense for medium sized caches but isn't ideal when caches become very large.

      That's because the volume of work a checkpoint needs to do is linearly related to the volume of dirty content currently in the cache.

      MongoDB has customers with caches of up to 512GB and over, and 20% of that value is 102GB. There is no utility in allowing such a large volume of dirty content in the cache for most workloads. Allowing that much dirty content makes checkpoints very expensive, which can cause operational issues for many workloads.

      We should consider either adding an absolute cap on the default dirty threshold, for example, once it reaches 10GB stop adhering to the 20% configuration.

      While we are doing this work, we should also consider whether 20% is a sensible value for small cache sizes (I think that is 2GB or less).

      We also need to ensure that eviction_updates_trigger, eviction_updates_target and eviction_dirty_target are consistent when changing the default behavior.

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: