Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-11368

Understanding eviction in an application that has a large number of updates

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 5.0.11
    • Component/s: None
    • Environment:
      MongoDB : 5.0.11
      OS : Ubuntu 20.04.4 LTS

      Hey team, I was going through the source code of the current `develop` branch (Git commit - `569cabb73`)

       

      Recently our production systems have been facing issue where % of update_bytes are frequently breaching 10% cache size limit and hence application threads are getting involved for eviction (in turn increasing operational latency, please refer to this ticket - https://www.mongodb.com/community/forums/t/why-is-mongodb-using-application-threads/234620

      Our current configurations are as follows : 

      • MongoDB 5.0.11 server in cluster mode with three shards, each of them having one primary and 2 secondary nodes.
      • Each node is configured with a maximum of 20 eviction threads.

       

      When checking the source code, I noticed that the __wt_cache data structures has a field bytes_updates which are the relevant bytes in this scenario.

      It's only been manipulated from b-tree's inline functions 
      {}wt_cache_page_inmem_incr, *__wt_cache_page_byte_updates_decr* etc and none of them are being accessed by eviction threads, hence it's possible that the following scenario occurs:

        * whenever a page is evicted from cache's queues, the update_bytes are not updated

      • Future cache eviction computation is performed on stale data.
      • Let's assume only new btrees are added (our use case with MongoDB is extremely write-heavy)
      • Then it's possible that eviction threads assume that the update bytes are only increasing even though pages have been evicted by the earlier worker threads and hence pull in application threads.
        Is this kind of behavior expected? Would it be helpful that cache eviction threads also update the byte-related fields in cache struct after evicting a page or am I missing something in my analysis?

      Also, is there any way we could circumvent the current production issue?

      Thank you!

        1. image-2023-07-31-22-50-59-638.png
          185 kB
          Uddeshya Singh
        2. image-2023-07-31-22-51-34-384.png
          168 kB
          Uddeshya Singh

            Assignee:
            yuan.fang@mongodb.com Yuan Fang
            Reporter:
            uddeshya.singh@gojek.com Uddeshya Singh
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: