Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-13794

Unused snapshot history consuming significant heap space

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.6.2, 2.7.1
    • Affects Version/s: 2.6.0
    • Component/s: Diagnostics
    • None
    • Fully Compatible
    • ALL
    • Server 2.7.1

      Issue Status as of May 16, 2014

      ISSUE SUMMARY
      mongod consumes significant heap space on unused snapshot history when snapshots are enabled. This increased memory use is not a leak, and is proportional to the number of collections.

      USER IMPACT
      Users will see increased memory usage by mongod processes.

      WORKAROUNDS
      N/A

      AFFECTED VERSIONS
      All 2.4 and 2.6 MongoDB production releases are affected by this isse.

      FIX VERSION
      The fix is included in the 2.6.2 production release.

      RESOLUTION DETAILS
      The snapshot thread, which provides historical collection level and lock statistics for use by the web interface, is only needed when HTTP is enabled. To avoid this extra memory consumption, disable the HTTP interface.

      Original description

      We saw a case where an upgrade from 2.4 to 2.6 led to non-mapped memory peak growing for all 2.6.0 members of a replica set. Taking the data set in question and 15 minute sampling of read operations I was able to reproduce 200MB growth. Heap profiling in Valgrind led to discovery that this memory was being consumed by the snapshots thread (db/stats/snapshots.h/.cpp).

      The snapshots thread currently maintains an array of 100 snapshots. Each snapshot contains collection level usage data, the size of which grows with each collection contained within a mongod instance. The data set used to reproduce contains 3588 collections.

      The growth occurs in the following valgrind massif stack. The full ms_print valgrind output will be attached as well.

      99.41% (153,881,596B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->55.57% (86,016,800B) 0x138BE28: mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::Area::Area(mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::Area const&) (unordered_fast_key_table_internal.h:34)
| ->55.57% (86,016,800B) 0x138BC40: mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::copyTo(mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>*) const (unordered_fast_key_table_internal.h:127)
|   ->55.57% (86,016,800B) 0x138BB85: mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::operator=(mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther> const&) (unordered_fast_key_table.h:94)
|     ->55.57% (86,016,800B) 0x138B7B1: mongo::StringMap<mongo::Top::CollectionData>::operator=(mongo::StringMap<mongo::Top::CollectionData> const&) (string_map.h:46)
|       ->55.57% (86,016,800B) 0x138AFCC: mongo::Top::cloneMap(mongo::StringMap<mongo::Top::CollectionData>&) const (top.cpp:130)
|         ->55.57% (86,016,800B) 0x13878C4: mongo::SnapshotData::takeSnapshot() (snapshots.cpp:47)
|           ->55.57% (86,016,800B) 0x1387E03: mongo::Snapshots::takeSnapshot() (snapshots.cpp:83)
|             ->55.57% (86,016,800B) 0x13881D6: mongo::SnapshotThread::run() (snapshots.cpp:128)
|               ->55.57% (86,016,800B) 0x151E2B8: mongo::BackgroundJob::jobBody() (background.cpp:147)
|                 ->55.57% (86,016,800B) 0x1521793: boost::_mfi::mf0<void, mongo::BackgroundJob>::operator()(mongo::BackgroundJob*) const (mem_fn_template.hpp:49)
|                   ->55.57% (86,016,800B) 0x15216F6: void boost::_bi::list1<boost::_bi::value<mongo::BackgroundJob*> >::operator()<boost::_mfi::mf0<void, mongo::BackgroundJob>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, mongo::BackgroundJob>&, boost::_bi::list0&, int) (bind.hpp:253)
|                     ->55.57% (86,016,800B) 0x15216A5: boost::_bi::bind_t<void, boost::_mfi::mf0<void, mongo::BackgroundJob>, boost::_bi::list1<boost::_bi::value<mongo::BackgroundJob*> > >::operator()() (bind_template.hpp:20)
|                       ->55.57% (86,016,800B) 0x152166A: boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf0<void, mongo::BackgroundJob>, boost::_bi::list1<boost::_bi::value<mongo::BackgroundJob*> > > >::run() (thread.hpp:62)
|                         ->55.57% (86,016,800B) 0x15C11E8: thread_proxy (thread.cpp:121)
|                           ->55.57% (86,016,800B) 0x4E36180: start_thread (pthread_create.c:312)
|                             ->55.57% (86,016,800B) 0x596730B: clone (clone.S:111)
|

        1. massif.out
          74 kB
          James Wahlin

            Assignee:
            james.wahlin@mongodb.com James Wahlin
            Reporter:
            james.wahlin@mongodb.com James Wahlin
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: