Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-14905

mongod crashes without any error message

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.6.4
    • Component/s: Stability
    • None
    • ALL

      On a 1GB RAM Ubuntu server running on Digital Ocean, a single-instance mongod crashes silently.

      uname -a:

      Linux stockbase 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

      last lines of /var/mongodb/log:

      2014-08-14T12:46:13.486-0700 [conn5] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001307617", url: "http://www.sec.gov/Archives/edgar/data/1307617/000130761714000004/0001307617-14-000004-index.htm", pubDate: new Date(1408045202000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:0 locks(micros) r:704798 reslen:48 704ms
2014-08-14T12:46:14.114-0700 [conn2] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001167507", url: "http://www.sec.gov/Archives/edgar/data/1167507/000114036114032723/0001140361-14-032723-index.htm", pubDate: new Date(1408045175000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:0 locks(micros) r:622573 reslen:48 622ms
2014-08-14T12:46:14.799-0700 [conn4] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000859804", url: "http://www.sec.gov/Archives/edgar/data/859804/000085980414000005/0000859804-14-000005-index.htm", pubDate: new Date(1408045167000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:1 locks(micros) r:1082196 reslen:48 679ms
2014-08-14T12:46:15.407-0700 [conn3] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001538375", url: "http://www.sec.gov/Archives/edgar/data/1538375/000104746914007004/0001047469-14-007004-index.htm", pubDate: new Date(1408045158000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:0 locks(micros) r:602606 reslen:48 602ms
2014-08-14T12:46:16.065-0700 [conn5] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001492224", url: "http://www.sec.gov/Archives/edgar/data/1492224/000108514614001748/0001085146-14-001748-index.htm", pubDate: new Date(1408045126000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:1 locks(micros) r:792784 reslen:48 653ms
2014-08-14T12:47:20.622-0700 [clientcursormon] mem (MB) res:281 virt:8691
2014-08-14T12:47:20.622-0700 [clientcursormon]  mapped (incl journal view):8254
2014-08-14T12:47:20.622-0700 [clientcursormon]  connections:10
2014-08-14T12:48:27.858-0700 [conn4] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000811831", url: "http://www.sec.gov/Archives/edgar/data/811831/000081183114000003/0000811831-14-000003-index.htm", pubDate: new Date(1408045304000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:33 locks(micros) r:5274901 reslen:48 3324ms
2014-08-14T12:48:28.303-0700 [conn5] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000943034", url: "http://www.sec.gov/Archives/edgar/data/943034/000089710114001151/0000897101-14-001151-index.htm", pubDate: new Date(1408045124000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:24 locks(micros) r:5982433 reslen:48 3454ms
2014-08-14T12:48:29.390-0700 [conn3] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000943034", url: "http://www.sec.gov/Archives/edgar/data/943034/000089710114001151/0000897101-14-001151-index.htm", pubDate: new Date(1408045124000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:56 locks(micros) r:6455049 reslen:48 4445ms
2014-08-14T13:41:23.639-0700 ***** SERVER RESTARTED *****

      I've run the application code again several times, and one time, the log file did show that mongod was running out of memory:

      2014-08-14T13:41:51.054-0700 [conn9] insert stockbase-prod.content query: { ct: "filing", formNumber: "6-K", filerName: "WOORI FINANCE HOLDINGS CO LTD", cik: "0001264136", filerType: "Filer", url: "http://www.sec.gov/Archives/edgar/data/1264136/000119312514310069/0001193125-14-310069-index.htm", summary: "<b>Filed:</b> 2014-08-14 <b>AccNo:</b> 0001193125-14-310069 <b>Size:</b> 4 MB", accessionNumber: "0001193125-14-310069", size: "4", pubDate: new Date(1408046048000), score: 0, type: "investor", _id: "SKKrhjDxgiuGzewkt" } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:12606 1324ms
2014-08-14T13:41:51.055-0700 [conn9] command stockbase-prod.$cmd command: insert { $msg: "query not recording (too large)" } keyUpdates:0 numYields:0 locks(micros) w:284 reslen:40 1334ms
2014-08-14T13:41:51.296-0700 [conn8] query stockbase-prod.feedParsing.jobs query: { $query: { type: { $in: [ "parseFeed" ] }, status: "ready", runId: null, after: { $lte: new Date(1408048907796) }, retries: { $gt: 0 } }, orderby: { priority: 1, retryUntil: 1, after: 1 } } planSummary: COLLSCAN, COLLSCAN ntoreturn:1 ntoskip:0 keyUpdates:0 numYields: 9 locks(micros) r:101699 nreturned:0 reslen:20 107ms
2014-08-14T13:41:57.421-0700 [conn10] Assertion: 10000:out of memory BufBuilder
2014-08-14T13:41:57.439-0700 [conn8] Assertion: 10000:out of memory BufBuilder
2014-08-14T13:41:57.446-0700 [conn9] Assertion: 10000:out of memory BufBuilder
2014-08-14T13:41:57.736-0700 [conn6] SEVERE: out of memory, printing stack and exiting:
0x11e6111 0x11e5408 0x1239bfd 0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6 0x7f7b7e3dedb5 0x7f7b7e3df077 0x11e61c4 0x11e5408 0x1239bfd
0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6
Unable to collect backtrace symbols (errno:12 Cannot allocate memory)
2014-08-14T13:41:57.736-0700 [conn7] SEVERE: out of memory, printing stack and exiting:
0x11e6111 0x11e5408 0x1239bfd 0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6 0x7f7b7e3dedb5 0x7f7b7e3df077 0x11e61c4 0x11e5408 0x1239bfd
0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6
Unable to collect backtrace symbols (errno:12 Cannot allocate memory)

      Would it be possible to dump to the log the cause of every crash?

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            dandv Dan Dascalescu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: