Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-6188

Memory leak in WiredTiger

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.0-rc4
    • Component/s: None
    • None

      This isn't new in rc4, it has been a problem in previous earlier release candidates, but rc4 is more stable then previous release candidates so it is easier to observe.

      The workload is Linkbench with the same configuration that gets WT-6175. The test database fits in memory (maxid=10000001 in test_control.yml). The test sequence is load, create index, run many 1 hour transaction tests. The size of the mongod process (VSZ and RSS) grows with each transaction test and after ~10 hours the mongod process runs out of memory. In this case the test server has 60G of RAM.

      The test, Linkbench2, is in DSI although that config has to be modified:
      1) set cacheSizeGB=40
      2) I don't think it matters but database files were not compressed
      3) run the transaction test for about 20 1-hour loops

      Here is an example of the size of mongod at the end of the first 19 1-hour loops. OOM happens on loop 20. cacheSizeGB=40 and the test server has ~60G of RAM

      ec2-user  64941  492 68.4 44794496 42301696 ?   SLl  15:49 637:09 bin/mongod --config mongo.conf
      ec2-user  64941  504 72.3 47211428 44722968 ?   SLl  15:49 956:21 bin/mongod --config mongo.conf
      ec2-user  64941  511 75.6 49300640 46792636 ?   SLl  15:49 1275:28 bin/mongod --config mongo.conf
      ec2-user  64941  515 79.3 51611064 49040132 ?   SLl  15:49 1594:18 bin/mongod --config mongo.conf
      ec2-user  64941  518 82.3 53831364 50924904 ?   SLl  15:49 1918:05 bin/mongod --config mongo.conf
      ec2-user  64941  522 85.7 55879612 53017960 ?   SLl  15:49 2244:04 bin/mongod --config mongo.conf
      ec2-user  64941  525 87.7 57198684 54228240 ?   SLl  15:49 2572:36 bin/mongod --config mongo.conf
      ec2-user  64941  528 89.3 58190036 55252252 ?   SLl  May11 2908:16 bin/mongod --config mongo.conf
      ec2-user  64941  533 93.2 60639740 57651416 ?   SLl  May11 3254:13 bin/mongod --config mongo.conf
      ec2-user  64941  537 93.7 61327952 57973468 ?   SLl  May11 3601:41 bin/mongod --config mongo.conf
      ec2-user  64941  540 93.4 61360724 57785736 ?   SLl  May11 3948:40 bin/mongod --config mongo.conf
      ec2-user  64941  544 94.1 61696636 58195144 ?   SLl  May11 4300:56 bin/mongod --config mongo.conf
      ec2-user  64941  549 95.0 62163636 58774912 ?   SLl  May11 4671:21 bin/mongod --config mongo.conf
      ec2-user  64941  553 95.9 62884620 59333032 ?   SLl  May11 5038:35 bin/mongod --config mongo.conf
      ec2-user  64941  557 95.6 62982936 59107644 ?   SLl  May11 5407:46 bin/mongod --config mongo.conf
      ec2-user  64941  560 95.5 62982936 59079712 ?   SLl  May11 5771:55 bin/mongod --config mongo.conf
      ec2-user  64941  563 95.2 63015708 58901768 ?   SLl  May11 6140:23 bin/mongod --config mongo.conf
      ec2-user  64941  565 95.0 63015708 58774420 ?   SLl  May11 6510:13 bin/mongod --config mongo.conf
      ec2-user  64941  568 95.1 63015708 58797536 ?   SLl  May11 6878:00 bin/mongod --config mongo.conf
      

        1. cursorleak.png
          208 kB
          Bruce Lucas
        2. freed.png
          141 kB
          Bruce Lucas
        3. ftdc.tar.gz
          32.21 MB
          Mark Callaghan
        4. leak.png
          150 kB
          Bruce Lucas
        5. metrics.2020-05-11T15-49-05Z-00000.r1
          10.00 MB
          Mark Callaghan
        6. metrics.2020-05-11T19-10-05Z-00000.r1
          10.00 MB
          Mark Callaghan
        7. metrics.2020-05-11T22-40-14Z-00000.r1
          10.00 MB
          Mark Callaghan
        8. metrics.2020-05-12T02-07-59Z-00000.r1
          9.98 MB
          Mark Callaghan
        9. metrics.2020-05-12T05-25-20Z-00000.r1
          9.97 MB
          Mark Callaghan
        10. metrics.2020-05-12T08-47-29Z-00000.r1
          9.98 MB
          Mark Callaghan
        11. metrics.2020-05-12T12-11-11Z-00000.r1
          2.34 MB
          Mark Callaghan
        12. metrics.interim.r1
          71 kB
          Mark Callaghan
        13. repro-fail.png
          85 kB
          Sulabh Mahajan
        14. stacks.png
          102 kB
          Bruce Lucas

            Assignee:
            sulabh.mahajan@mongodb.com Sulabh Mahajan
            Reporter:
            mark.callaghan@mongodb.com Mark Callaghan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: