Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9869

CLONE - Mongo DB crashing with an error: Not enough storage is available to process this command

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Blocker - P1 Blocker - P1
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Environment:
      Windows server

      Mongo DB is crashing which is causing the application to fail. When extracted mongo Db logs found below error 

      2022-09-11T00:36:21.977+0000 E STORAGE [WTJournalFlusher] WiredTiger (-28992) [1662856581:400819][7072:1995194080], WT_SESSION.log_flush: journal/WiredTigerLog.0000158373 handle-sync: FlushFileBuffers error: Not enough storage is available to process this command.

      2022-09-11T00:36:21.989+0000 I - [WTJournalFlusher] Invariant failure: s->log_flush(s, "sync=on") resulted in status UnknownError: -28992: Not enough storage is available to process this command.

      After restarting application servers mongo started working as expected. But this is the second occurrence and we want to know for possible solutions to fix this issue to avoid future occurrences.

      Disk has enough space :  800gb in D drive, 40-50gb is C drive. And mongo DB is around 720gb

      Mongo DB version used : 3.2.6

      Operating system is : windows.

       

      2022-09-11T00:36:21.849+0000 I COMMAND  [conn3987] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

      { r: 1, w: 1 }

      }, Database: { acquireCount:

      { w: 1 }

      }, Collection: { acquireCount:

      { w: 1 }

      } } 113ms
      2022-09-11T00:36:21.977+0000 E STORAGE  [WTJournalFlusher] WiredTiger (-28992) [1662856581:400819][7072:1995194080], WT_SESSION.log_flush: journal/WiredTigerLog.0000158373 handle-sync: FlushFileBuffers error: Not enough storage is available to process this command.
      2022-09-11T00:36:21.989+0000 I -        [WTJournalFlusher] Invariant failure: s->log_flush(s, "sync=on") resulted in status UnknownError: -28992: Not enough storage is available to process this command.
       at src\mongo\db\storage\wiredtiger\wiredtiger_session_cache.cpp 203
      2022-09-11T00:36:25.065+0000 I COMMAND  [conn4275] update meolutdb.eMSStatus query: { _id: "421d0553-e28b-4feb-a48e-7bfb6ed129fc" } update: { _id: "421d0553-e28b-4feb-a48e-7bfb6ed129fc", _class: "com.emsgt.lut.il.dal.api.EMSStatus", subsystem_id: 3, service_id: 10005, service_description: "Motor Temperature", timestamp: 1662856584631000000, status: "OK", errors: " 34.00 celsius", subsystem_type: "Antenna" } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 keyUpdates:6 writeConflicts:0 numYields:1 locks:{ Global: { acquireCount:

      { r: 2, w: 2 }

      }, Database: { acquireCount:

      { w: 2 }

      }, Collection: { acquireCount:

      { w: 2 }

      } } 113ms
      2022-09-11T00:36:26.505+0000 I COMMAND  [conn3965] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

      { r: 1, w: 1 }

      }, Database: { acquireCount:

      { w: 1 }

      }, Collection: { acquireCount:

      { w: 1 }

      } } 119ms
      2022-09-11T00:36:26.632+0000 I COMMAND  [conn4001] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

      { r: 1, w: 1 }

      }, Database: { acquireCount:

      { w: 1 }

      }, Collection: { acquireCount:

      { w: 1 }

      } } 156ms
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(174)                                   mongo::printStackTrace+0x43
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\log.cpp(136)                                                  mongo::logContext+0xa8
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\assert_util.cpp(164)                                          mongo::invariantOKFailed+0x14c
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_session_cache.cpp(203)            mongo::WiredTigerSessionCache::waitUntilDurable+0x2bd
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(97)                 mongo::WiredTigerKVEngine::WiredTigerJournalFlusher::run+0x1a8
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\background.cpp(152)                                           mongo::BackgroundJob::jobBody+0x1b1
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    c:\program files (x86)\microsoft visual studio 12.0\vc\include\thr\xthread(188)  std::LaunchPad<std::_Bind<0,void,std::_Bind<1,void,std::_Pmf_wrap<void (_cdecl mongo::dur::JournalWriter::*)(void) __ptr64,void,mongo::dur::JournalWriter>,mongo::dur::JournalWriter * __ptr64 const> > >::_Go+0x1c
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\stdcpp\thr\threadcall.cpp(28)                           _Call_func+0x14
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(376)                                 _callthreadstartex+0x17
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(354)                                 _threadstartex+0x102
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] kernel32.dll                                                                                   BaseThreadInitThunk+0xd
      2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] 
      2022-09-11T00:36:50.126+0000 I -        [WTJournalFlusher] 

      ***aborting after invariant() failure

       

        1. mongodb_LP_2022-08-22T20-46-44.log
          9.75 MB
        2. mongodb_LP (2).log
          137.95 MB

            Assignee:
            Unassigned Unassigned
            Reporter:
            k.mamatha23@gmail.com Mamatha K
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: