Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34377

Possible memory leak

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.3
    • Component/s: WiredTiger
    • None
    • Environment:
      HW: AWS c4.2xlarge
      OS: Ubuntu Xenila
      Kernel: 4.4.0-92-generic
    • ALL
    • Storage Non-NYC 2018-05-07

      After upgrading one of our mongodb cluster to 3.6.3 I noticed that mongodb is leaking memory over time or something (maybe it was leaking also in 3.4)
      wiredTiger stats reports only 5GB of cache used, but according to ps mongod is using more than 12GB

      I haven't restarted this host yet, I can collect more information if needed.

      USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
      mongodb   3021 97.1 84.3 21641472 12996084 ?   Ssl  Feb27 56667:27 /usr/bin/mongod --config /etc/mongod.conf
      

      wiredTiger

      db.serverStatus().wiredTiger.cache
      {
      	"application threads page read from disk to cache count" : 2657042,
      	"application threads page read from disk to cache time (usecs)" : 388085328,
      	"application threads page write from cache to disk count" : 266761274,
      	"application threads page write from cache to disk time (usecs)" : 2414299779,
      	"bytes belonging to page images in the cache" : 3403211247,
      	"bytes belonging to the lookaside table in the cache" : 182,
      	"bytes currently in the cache" : 5435745914,
      	"bytes not belonging to page images in the cache" : 2032534667,
      	"bytes read into cache" : 47026988836,
      	"bytes written from cache" : NumberLong("3776856175328"),
      	"checkpoint blocked page eviction" : 1708,
      	"eviction calls to get a page" : 32023358,
      	"eviction calls to get a page found queue empty" : 2515062,
      	"eviction calls to get a page found queue empty after locking" : 509387,
      	"eviction currently operating in aggressive mode" : 0,
      	"eviction empty score" : 0,
      	"eviction passes of a file" : 1883881,
      	"eviction server candidate queue empty when topping up" : 85529,
      	"eviction server candidate queue not empty when topping up" : 859553,
      	"eviction server evicting pages" : 0,
      	"eviction server slept, because we did not make progress with eviction" : 31345553,
      	"eviction server unable to reach eviction goal" : 0,
      	"eviction state" : 32,
      	"eviction walk target pages histogram - 0-9" : 129324,
      	"eviction walk target pages histogram - 10-31" : 795080,
      	"eviction walk target pages histogram - 128 and higher" : 0,
      	"eviction walk target pages histogram - 32-63" : 126038,
      	"eviction walk target pages histogram - 64-128" : 833439,
      	"eviction walks abandoned" : 252649,
      	"eviction walks gave up because they restarted their walk twice" : 286047,
      	"eviction walks gave up because they saw too many pages and found no candidates" : 17506,
      	"eviction walks gave up because they saw too many pages and found too few candidates" : 32906,
      	"eviction walks reached end of tree" : 914012,
      	"eviction walks started from root of tree" : 580255,
      	"eviction walks started from saved location in tree" : 1303626,
      	"eviction worker thread active" : 4,
      	"eviction worker thread created" : 0,
      	"eviction worker thread evicting pages" : 28921953,
      	"eviction worker thread removed" : 0,
      	"eviction worker thread stable number" : 0,
      	"failed eviction of pages that exceeded the in-memory maximum count" : 35487,
      	"failed eviction of pages that exceeded the in-memory maximum time (usecs)" : 517310,
      	"files with active eviction walks" : 0,
      	"files with new eviction walks started" : 627965,
      	"force re-tuning of eviction workers once in a while" : 0,
      	"hazard pointer blocked page eviction" : 57852,
      	"hazard pointer check calls" : 33233880,
      	"hazard pointer check entries walked" : 190015256,
      	"hazard pointer maximum array length" : 3,
      	"in-memory page passed criteria to be split" : 235698,
      	"in-memory page splits" : 86165,
      	"internal pages evicted" : 8326,
      	"internal pages split during eviction" : 61,
      	"leaf pages split during eviction" : 112849,
      	"lookaside score" : 0,
      	"lookaside table entries" : 0,
      	"lookaside table insert calls" : 0,
      	"lookaside table remove calls" : 0,
      	"maximum bytes configured" : 7347372032,
      	"maximum page size at eviction" : 0,
      	"modified pages evicted" : 28692743,
      	"modified pages evicted by application threads" : 0,
      	"overflow pages read into cache" : 0,
      	"page split during eviction deepened the tree" : 0,
      	"page written requiring lookaside records" : 0,
      	"pages currently held in the cache" : 176311,
      	"pages evicted because they exceeded the in-memory maximum count" : 105360,
      	"pages evicted because they exceeded the in-memory maximum time (usecs)" : 965601663,
      	"pages evicted because they had chains of deleted items count" : 145901,
      	"pages evicted because they had chains of deleted items time (usecs)" : 293750697,
      	"pages evicted by application threads" : 0,
      	"pages queued for eviction" : 94503526,
      	"pages queued for urgent eviction" : 302231,
      	"pages queued for urgent eviction during walk" : 16993,
      	"pages read into cache" : 2660205,
      	"pages read into cache requiring lookaside entries" : 0,
      	"pages read into cache skipping older lookaside entries" : 0,
      	"pages read into cache with skipped lookaside entries needed later" : 0,
      	"pages requested from the cache" : 444638466854,
      	"pages seen by eviction walk" : 675910019,
      	"pages selected for eviction unable to be evicted" : 103201,
      	"pages walked for eviction" : 866578969,
      	"pages written from cache" : 295217700,
      	"pages written requiring in-memory restoration" : 886952,
      	"percentage overhead" : 8,
      	"tracked bytes belonging to internal pages in the cache" : 17646012,
      	"tracked bytes belonging to leaf pages in the cache" : 5418099902,
      	"tracked dirty bytes in the cache" : 29123058,
      	"tracked dirty pages in the cache" : 74,
      	"unmodified pages evicted" : 4344899
      }
      

        1. Screen Shot 2018-04-08 at 20.22.20.png
          Screen Shot 2018-04-08 at 20.22.20.png
          425 kB
        2. increase.png
          increase.png
          67 kB
        3. diagnostic.data.tar
          105.13 MB

            Assignee:
            donald.anderson@mongodb.com Donald Anderson
            Reporter:
            tarvip Tarvi Pillessaar
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: