Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27909

OOM and possible memory leaks

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.4.1
    • Component/s: WiredTiger
    • ALL

      We have switched one of our servers to WT to test it out.
      Mongo version: 3.4.1
      aws: i2.2xlarge
      FS: ext4
      Swap: off
      daemon: mongod

      settings:

      systemLog: 
          destination: file
          path: "/var/log/mongodb/mongodb.log"
          quiet: false
          logAppend: true
          logRotate: reopen
      net: 
          http: 
              enabled: true
              RESTInterfaceEnabled: true
          port: 27017
      operationProfiling: 
          mode: slowOp
          slowOpThresholdMs: 1000
      
      storage: 
          dbPath: /ssd/var/lib/mongodb
      
          engine: wiredTiger
      

      after 24 hours it eated all server memory failed with OOM:

      Unable to find source-code formatter for language: text. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      2017-02-02T11:08:44.353+0000 I NETWORK  [thread1] connection accepted from xxx.xxx.xxx.xxx:41681 #27019 (1611 connections now open)
      2017-02-02T11:08:44.927+0000 F -        [statsSnapshot] out of memory.
      
       0x7f02698f3221 0x7f02698f2734 0x7f0269ab02cb 0x7f026a3e9d44 0x7f02694f2df2 0x7f02694ec34e 0x7f02694ec95c 0x7f02698629fd 0x7f026a3681d0 0x7f0266fd6184 0x7f0266d0337d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"7F0268374000","o":"157F221","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F0268374000","o":"157E734","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"7F0268374000","o":"173C2CB"},{"b":"7F0268374000","o":"2075D44","s":"tc_new"},{"b":"7F0268374000","o":"117EDF2","s":"_ZNK5mongo3Top8cloneMapERNS_21UnorderedFastKeyTableINS_10StringDataENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_14CollectionDataENS_15StringMapTraitsEEE"},{"b":"7F0268374000","o":"117834E","s":"_ZN5mongo9Snapshots12takeSnapshotEv"},{"b":"7F0268374000","o":"117895C","s":"_ZN5mongo19StatsSnapshotThread3runEv"},{"b":"7F0268374000","o":"14EE9FD","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"7F0268374000","o":"1FF41D0"},{"b":"7F0266FCE000","o":"8184"},{"b":"7F0266C09000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.1", "gitVersion" : "5e103c4f5583e2566a45d740225dc250baacfbd7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-29-generic", "version" : "#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "b" : "7F0268374000", "elfType" : 3, "buildId" : "B0B5767DC6B7AB9D129191D0FA1F55666B34170F" }, { "b" : "7FFFEDCFE000", "elfType" : 3, "buildId" : "3D068D088E7EAC15D9DA7C3AC912E783C0897EE7" }, { "b" : "7F0267EF0000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "15A6A69B862FAC0AA589921DB1B0DF8A673FEC81" }, { "b" : "7F0267B14000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "607C4A6F3428B6F065D55691AFDC4E343786A8A5" }, { "b" : "7F026790C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F0267708000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f02698f3221] 
       mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x7f02698f2734] 
       mongod(+0x173C2CB) [0x7f0269ab02cb]
       mongod(tc_new+0x224) [0x7f026a3e9d44]
       mongod(_ZNK5mongo3Top8cloneMapERNS_21UnorderedFastKeyTableINS_10StringDataENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_14CollectionDataENS_15StringMapTraitsEEE+0x62) [0x7f02694f2df2]
       mongod(_ZN5mongo9Snapshots12takeSnapshotEv+0x4E) [0x7f02694ec34e]
       mongod(_ZN5mongo19StatsSnapshotThread3runEv+0x9C) [0x7f02694ec95c] 
       mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x16D) [0x7f02698629fd] 
       mongod(+0x1FF41D0) [0x7f026a3681d0]
       libpthread.so.0(+0x8184) [0x7f0266fd6184] 
       libc.so.6(clone+0x6D) [0x7f0266d0337d]
      -----  END BACKTRACE  -----
      2017-02-02T11:20:05.538+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
      

      I restarted mongo server and added a swap - 10GB.. for some time memory usage was at about 65% and then started to go up again.
      After 24h I have this picture:

      Unable to find source-code formatter for language: plain. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      root@xxxxxx:/var/log/mongodb# free -h
                   total       used       free     shared    buffers     cached
      Mem:           29G        29G       227M        36K       2.4M        62M
      -/+ buffers/cache:        29G       292M
      Swap:           9G       2.5G       7.5G
      

      So all memory eaten and roughly 2.5GB of swap occupied.
      I don't think it is correct nor expected behavior. Mongo should not eat more memory than available.
      Server still running and Im trying to collect serverStatus info every 30 mins now.

      Please find in attachment last 4 serverStatus reports.
      With MMAPv1 we never had usage more than 15-20%, not saying that we never seen OOM.

        1. mongo-mem.png
          mongo-mem.png
          15 kB
        2. status_20170203_1730.json
          20 kB
        3. status_20170203_1800.json
          20 kB
        4. status_20170203_1830.json
          20 kB
        5. status_20170203_1900.json
          20 kB
        6. WTcursors.png
          WTcursors.png
          139 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            dyatlov Vitaly Dyatlov
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: