Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6519

MongoDB Crash Under High Load

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Blocker - P1 Blocker - P1
    • None
    • Affects Version/s: 2.0.1, 2.0.3, 2.0.4, 2.0.6
    • Component/s: Storage
    • None
    • Environment:
      ubuntu 10.04 64-bit
    • ALL

      The DB was configured as a capped collection. From the beginning of 2012 to now, the same kind of crash happened about 10 times. (3 times this week and today crashed 2 times). Cause it's used as log db, journal is disabled (--nojournal)

      The crash log looks like the following:

      Mon Jul 16 15:26:58 [conn5573] query track.entries ntoreturn:1 idhack:1 reslen:348 105ms
      Mon Jul 16 15:26:58 [conn6107] query track.entries ntoreturn:1 idhack:1 reslen:344 104ms
      Mon Jul 16 15:26:58 [conn5575] query track.entries ntoreturn:1 idhack:1 reslen:263 165ms
      Mon Jul 16 15:26:59 [conn7058] insert track.entries 254ms
      Mon Jul 16 15:27:05 [conn7052] insert track.entries 124ms
      Mon Jul 16 15:27:08 [conn5567] query track.entries ntoreturn:1 idhack:1 reslen:323 368ms
      Mon Jul 16 15:27:09 [conn7051] insert track.entries 211ms
      Mon Jul 16 15:27:19 [conn7052] insert track.entries 104ms
      Mon Jul 16 15:27:26 [conn7055] insert track.entries 182ms
      Mon Jul 16 15:27:26 Invalid access at address: 0x7eeb90b61984

      Mon Jul 16 15:27:26 Got signal: 11 (Segmentation fault).

      Mon Jul 16 15:27:26 Backtrace:
      0xa90d79 0xa91350 0x7f479d1df8f0 0x885d49 0x888cc2 0x88c347 0x88ebb5 0xaa37d6 0x637497 0x7f479d1d69ca 0x7f479c78570d
      mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa90d79]
      mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa91350]
      /lib/libpthread.so.0(+0xf8f0) [0x7f479d1df8f0]
      mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x49) [0x885d49]
      mongod(_ZN5mongo11insertMultiERNS_9DbMessageEPKcRKNS_7BSONObjE+0x52) [0x888cc2]
      mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x327) [0x88c347]
      mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x11c5) [0x88ebb5]
      mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa37d6]
      mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637497]
      /lib/libpthread.so.0(+0x69ca) [0x7f479d1d69ca]
      /lib/libc.so.6(clone+0x6d) [0x7f479c78570d]

      Mon Jul 16 15:27:26 dbexit:
      Mon Jul 16 15:27:26 [conn7055] shutdown: going to close listening sockets...
      Mon Jul 16 15:27:26 [conn7055] closing listening socket: 5
      Mon Jul 16 15:27:26 [conn7055] closing listening socket: 6
      Mon Jul 16 15:27:26 [conn7055] closing listening socket: 7
      Mon Jul 16 15:27:26 [conn7055] removing socket file: /tmp/mongodb-27019.sock
      Mon Jul 16 15:27:26 [conn6300] got request after shutdown()
      Mon Jul 16 15:27:26 [conn6479] got request after shutdown()
      Mon Jul 16 15:27:26 [conn7055] shutdown: going to flush diaglog...
      Mon Jul 16 15:27:26 [conn7055] shutdown: going to close sockets...
      Mon Jul 16 15:27:26 [conn6750] got request after shutdown()
      Mon Jul 16 15:27:26 [conn7055] shutdown: waiting for fs preallocator...
      Mon Jul 16 15:27:26 [conn7055] shutdown: closing all files...
      Mon Jul 16 15:27:26 [conn6792] end connection 10.18.10.22:58600
      Mon Jul 16 15:27:26 [conn6640] end connection 10.18.10.25:28257
      Mon Jul 16 15:27:26 [conn6618] end connection 10.18.10.33:48380
      Mon Jul 16 15:27:26 [conn7060] end connection 10.18.10.32:35593
      Mon Jul 16 15:27:26 [conn6279] end connection 10.18.10.25:57854
      Mon Jul 16 15:27:26 [conn7057] end connection 10.18.10.32:35585
      Mon Jul 16 15:27:26 [conn7140] end connection 10.18.10.22:59207
      Mon Jul 16 15:27:26 [conn6650] end connection 10.18.10.23:51100
      Mon Jul 16 15:27:26 [conn7120] end connection 10.18.10.206:39333
      Mon Jul 16 15:27:26 [conn7054] end connection 10.18.10.32:35580
      Mon Jul 16 15:27:26 [conn7051] end connection 10.18.10.32:35576
      Mon Jul 16 15:27:26 [conn6544] end connection 10.18.10.48:18576
      Mon Jul 16 15:27:26 [conn7058] end connection 10.18.10.32:35586
      Mon Jul 16 15:27:26 [conn7052] end connection 10.18.10.32:35578
      Mon Jul 16 15:27:26 [conn5893] end connection 10.18.10.37:59303
      Mon Jul 16 15:27:26 [conn5937] end connection 10.18.10.32:44866
      Mon Jul 16 15:27:26 [conn7053] end connection 10.18.10.32:35579
      Mon Jul 16 15:27:26 [conn7059] end connection 10.18.10.32:35587
      Mon Jul 16 15:27:26 [conn7056] end connection 10.18.10.32:35584
      Mon Jul 16 15:27:26 [conn6820] end connection 10.18.10.30:52364
      Mon Jul 16 15:27:26 [conn6925] end connection 10.18.10.34:43771
      Mon Jul 16 15:27:26 [conn7050] end connection 10.18.10.32:35572
      Mon Jul 16 15:27:26 [conn6899] end connection 10.18.10.23:58106
      Mon Jul 16 15:27:26 [conn5863] end connection 10.18.10.33:49230
      Mon Jul 16 15:27:26 [conn5965] end connection 10.18.10.37:57746
      Mon Jul 16 15:27:26 [conn6919] end connection 10.18.10.25:21350
      Mon Jul 16 15:27:26 [conn5869] end connection 10.18.10.33:49301
      Mon Jul 16 15:27:26 [conn6684] end connection 10.18.10.23:56587
      Mon Jul 16 15:27:26 [conn6823] end connection 10.18.10.30:52380
      Mon Jul 16 15:27:26 [conn6887] end connection 10.18.10.28:56480
      Mon Jul 16 15:27:26 [conn6875] end connection 10.18.10.23:42136
      Mon Jul 16 15:27:26 [conn5977] end connection 10.18.10.32:59654
      Mon Jul 16 15:27:26 [conn6931] end connection 10.18.10.25:35808
      Mon Jul 16 15:27:26 [conn7062] end connection 10.18.10.30:40040
      Mon Jul 16 15:27:26 [conn6409] end connection 10.18.10.31:56892
      Mon Jul 16 15:27:26 [conn6123] end connection 10.18.10.43:34244
      Mon Jul 16 15:27:26 [conn5881] end connection 10.18.10.37:59271
      Mon Jul 16 15:27:26 [conn5930] end connection 10.18.10.32:44780
      Mon Jul 16 15:27:26 [conn6917] end connection 10.18.10.34:42988
      Mon Jul 16 15:27:26 [conn5261] end connection 10.18.10.43:60205
      Mon Jul 16 15:27:26 [conn6059] end connection 10.18.10.25:3087
      Mon Jul 16 15:27:26 [conn6554] end connection 10.18.10.32:46072
      Mon Jul 16 15:27:26 [conn5941] end connection 10.18.10.32:44924
      Mon Jul 16 15:27:26 [conn5569] end connection 10.18.10.31:48263
      Mon Jul 16 15:27:27 [conn5577] end connection 10.18.10.31:48346
      Mon Jul 16 15:27:27 [conn5567] end connection 10.18.10.31:48223
      154/187 82%
      Mon Jul 16 15:27:28 [conn5931] end connection 10.18.10.32:44781
      186/187 99%
      Mon Jul 16 15:27:30 [conn7055] closeAllFiles() finished
      Mon Jul 16 15:27:30 [conn7055] shutdown: removing fs lock...
      Mon Jul 16 15:27:30 dbexit: really exiting now
      Logstream::get called in uninitialized state
      Mon Jul 16 15:27:30 ERROR: Client::~Client _context should be null but is not; client:conn


      Thu Jul 19 11:08:16 [conn925] insert track.entries 447ms
      Thu Jul 19 11:08:24 [conn918] insert track.entries 449ms
      Thu Jul 19 11:08:27 [conn921] insert track.entries 110ms
      Thu Jul 19 11:08:32 [conn917] insert track.entries 234ms
      Thu Jul 19 11:08:32 [conn924] insert track.entries 121ms
      Thu Jul 19 11:08:36 [conn920] insert track.entries 126ms
      Thu Jul 19 11:08:37 [conn918] insert track.entries 123ms
      Thu Jul 19 11:08:38 [conn917] insert track.entries 450ms
      Thu Jul 19 11:08:46 [conn916] insert track.entries 454ms
      Thu Jul 19 11:08:53 [conn925] insert track.entries 156ms
      Thu Jul 19 11:08:55 [conn925] insert track.entries 431ms
      Thu Jul 19 11:09:00 [conn916] insert track.entries 299ms
      Thu Jul 19 11:09:03 [conn917] insert track.entries 230ms
      Thu Jul 19 11:09:07 [conn925] insert track.entries 109ms
      Thu Jul 19 11:09:09 [conn917] insert track.entries 447ms
      Thu Jul 19 11:09:13 [conn925] insert track.entries 619ms
      Thu Jul 19 11:09:16 [conn921] insert track.entries 234ms
      Thu Jul 19 11:09:21 [conn918] insert track.entries 122ms
      Thu Jul 19 11:09:24 [conn922] insert track.entries 440ms
      Thu Jul 19 11:09:28 [conn921] insert track.entries 182ms
      Thu Jul 19 11:09:32 [conn921] insert track.entries 578ms
      Thu Jul 19 11:09:40 [conn921] insert track.entries 479ms
      Thu Jul 19 11:09:43 [conn918] insert track.entries 191ms
      Thu Jul 19 11:09:48 [conn918] insert track.entries 191ms
      Thu Jul 19 11:09:48 [conn924] insert track.entries 414ms
      Thu Jul 19 11:09:57 [conn924] insert track.entries 452ms
      Thu Jul 19 11:09:58 [conn921] insert track.entries 265ms
      Thu Jul 19 11:10:05 [conn918] insert track.entries 463ms
      Thu Jul 19 11:10:13 [conn925] insert track.entries 449ms
      Thu Jul 19 11:10:21 [conn925] insert track.entries 562ms
      Thu Jul 19 11:10:23 [conn918] insert track.entries 135ms
      Thu Jul 19 11:10:23 [conn918] insert track.entries 357ms
      Thu Jul 19 11:10:29 [conn918] insert track.entries 458ms
      Thu Jul 19 11:10:30 [clientcursormon] mem (MB) res:44952 virt:375706 mapped:374809
      Thu Jul 19 11:10:33 [conn918] insert track.entries 319ms
      Thu Jul 19 11:10:38 [conn924] insert track.entries 453ms
      Thu Jul 19 11:10:46 [conn919] insert track.entries 426ms
      Thu Jul 19 11:10:54 [conn925] insert track.entries 254ms
      Thu Jul 19 11:10:58 [conn924] insert track.entries 119ms
      Thu Jul 19 11:11:00 [conn923] insert track.entries 444ms
      Thu Jul 19 11:11:03 [conn918] insert track.entries 346ms
      Thu Jul 19 11:11:08 [conn918] insert track.entries 471ms
      Thu Jul 19 11:11:12 [conn917] insert track.entries 102ms
      Thu Jul 19 11:11:16 [conn919] insert track.entries 237ms
      Thu Jul 19 11:11:20 [conn922] insert track.entries 115ms
      Thu Jul 19 11:11:23 [conn922] insert track.entries 130ms
      Thu Jul 19 11:11:23 [conn919] insert track.entries 507ms
      Thu Jul 19 11:11:27 [conn919] insert track.entries 130ms
      Thu Jul 19 11:11:28 [conn925] insert track.entries 318ms
      Thu Jul 19 11:11:32 [conn925] insert track.entries 441ms
      Thu Jul 19 11:11:33 [conn924] insert track.entries 168ms
      Thu Jul 19 11:11:40 [conn924] insert track.entries 238ms
      Thu Jul 19 11:11:44 [conn917] insert track.entries 110ms
      Thu Jul 19 11:11:46 [conn925] insert track.entries 449ms
      Thu Jul 19 11:11:55 [conn918] insert track.entries 427ms
      Thu Jul 19 11:11:58 [conn921] insert track.entries 102ms
      Thu Jul 19 11:11:58 [conn925] insert track.entries 162ms
      Thu Jul 19 11:12:03 [conn924] insert track.entries 198ms
      Thu Jul 19 11:12:03 [conn918] insert track.entries 126ms
      Thu Jul 19 11:12:03 [conn924] insert track.entries 466ms
      Thu Jul 19 11:12:13 [conn916] insert track.entries 922ms
      Thu Jul 19 11:12:21 [conn918] insert track.entries 250ms
      Thu Jul 19 11:12:25 [conn918] insert track.entries 121ms
      Thu Jul 19 11:12:28 [conn917] insert track.entries 720ms
      Thu Jul 19 11:12:36 [conn918] insert track.entries 480ms
      Thu Jul 19 11:12:38 [conn924] insert track.entries 238ms
      Thu Jul 19 11:12:43 [conn918] insert track.entries 261ms
      Thu Jul 19 11:12:45 [conn920] insert track.entries 223ms
      Thu Jul 19 11:12:45 Invalid access at address: 0x7ed9e8e71fb3

      Thu Jul 19 11:12:45 Got signal: 11 (Segmentation fault).

      Thu Jul 19 11:12:45 Backtrace:
      0xa90d79 0xa91350 0x7f35cd0278f0 0x885d49 0x888cc2 0x88c347 0x88ebb5 0xaa37d6 0x637497 0x7f35cd01e9ca 0x7f35cc5cd70d
      mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa90d79]
      mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa91350]
      /lib/libpthread.so.0(+0xf8f0) [0x7f35cd0278f0]
      mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x49) [0x885d49]
      mongod(_ZN5mongo11insertMultiERNS_9DbMessageEPKcRKNS_7BSONObjE+0x52) [0x888cc2]
      mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x327) [0x88c347]
      mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x11c5) [0x88ebb5]
      mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa37d6]
      mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637497]
      /lib/libpthread.so.0(+0x69ca) [0x7f35cd01e9ca]
      /lib/libc.so.6(clone+0x6d) [0x7f35cc5cd70d]

      Thu Jul 19 11:12:45 dbexit:
      Thu Jul 19 11:12:45 [conn925] shutdown: going to close listening sockets...
      Thu Jul 19 11:12:45 [conn925] closing listening socket: 5
      Thu Jul 19 11:12:45 [conn925] closing listening socket: 6
      Thu Jul 19 11:12:45 [conn925] closing listening socket: 7
      Thu Jul 19 11:12:45 [conn925] removing socket file: /tmp/mongodb-27019.sock
      Thu Jul 19 11:12:45 [conn925] shutdown: going to flush diaglog...
      Thu Jul 19 11:12:45 [conn925] shutdown: going to close sockets...
      Thu Jul 19 11:12:45 [conn925] shutdown: waiting for fs preallocator...
      Thu Jul 19 11:12:45 [conn925] shutdown: closing all files...
      Thu Jul 19 11:12:45 [conn1907] end connection 10.18.10.23:49164
      Thu Jul 19 11:12:45 [conn1623] end connection 10.18.10.48:52550
      Thu Jul 19 11:12:45 [conn564] end connection 10.18.10.30:48889
      Thu Jul 19 11:12:45 [conn650] end connection 10.18.10.43:46034
      Thu Jul 19 11:12:45 [conn1885] end connection 10.18.10.48:55198
      Thu Jul 19 11:12:45 [conn915] end connection 10.18.10.32:39604
      Thu Jul 19 11:12:45 [conn923] end connection 10.18.10.32:39615
      Thu Jul 19 11:12:45 [conn920] end connection 10.18.10.32:39611
      Thu Jul 19 11:12:45 [conn999] end connection 10.18.10.211:39177
      Thu Jul 19 11:12:45 [conn922] end connection 10.18.10.32:39614
      Thu Jul 19 11:12:45 [conn1515] end connection 10.18.10.48:51342
      Thu Jul 19 11:12:45 [conn916] end connection 10.18.10.32:39608
      Thu Jul 19 11:12:45 [conn921] end connection 10.18.10.32:39613
      Thu Jul 19 11:12:45 [conn1905] end connection 10.18.10.206:57120
      Thu Jul 19 11:12:45 [conn1645] end connection 10.18.10.32:35880
      Thu Jul 19 11:12:45 [conn1863] end connection 10.18.10.33:43729
      Thu Jul 19 11:12:45 [conn1543] end connection 10.18.10.211:50719
      Thu Jul 19 11:12:45 [conn991] end connection 10.18.10.211:39096
      Thu Jul 19 11:12:45 [conn710] end connection 10.18.10.34:32862
      Thu Jul 19 11:12:45 [conn756] end connection 10.18.10.28:58261
      Thu Jul 19 11:12:45 [conn975] end connection 10.18.10.211:38886
      Thu Jul 19 11:12:45 [conn977] end connection 10.18.10.211:38925
      Thu Jul 19 11:12:45 [conn1889] end connection 10.18.10.34:47631
      Thu Jul 19 11:12:45 [conn1795] end connection 10.18.10.43:47799
      Thu Jul 19 11:12:45 [conn1669] end connection 10.18.10.82:34125
      Thu Jul 19 11:12:45 [conn1043] end connection 10.18.10.210:51066
      Thu Jul 19 11:12:45 [conn624] end connection 10.18.10.43:45857
      Thu Jul 19 11:12:45 [conn784] end connection 10.18.10.37:33645
      Thu Jul 19 11:12:45 [conn648] end connection 10.18.10.48:48091
      Thu Jul 19 11:12:45 [conn628] end connection 10.18.10.43:45923
      Thu Jul 19 11:12:45 [conn1677] end connection 10.18.10.48:53089
      Thu Jul 19 11:12:45 [conn642] end connection 10.18.10.48:48029
      Thu Jul 19 11:12:45 [conn1051] end connection 10.18.10.210:51182
      Thu Jul 19 11:12:45 [conn632] end connection 10.18.10.43:45982
      Thu Jul 19 11:12:45 [conn620] end connection 10.18.10.43:45839
      Thu Jul 19 11:12:45 [conn1521] end connection 10.18.10.48:51546
      Thu Jul 19 11:12:45 [conn640] end connection 10.18.10.43:46008
      Thu Jul 19 11:12:45 [conn1041] end connection 10.18.10.210:51050
      Thu Jul 19 11:12:45 [conn720] end connection 10.18.10.206:49227
      Thu Jul 19 11:12:45 [conn1031] end connection 10.18.10.210:50910
      Thu Jul 19 11:12:45 [conn614] end connection 10.18.10.43:45809
      Thu Jul 19 11:12:45 [conn728] end connection 10.18.10.206:49320
      Thu Jul 19 11:12:45 [conn1655] end connection 10.18.10.82:33969
      Thu Jul 19 11:12:45 [conn714] end connection 10.18.10.34:32900
      Thu Jul 19 11:12:45 [conn1217] end connection 10.18.10.81:55458
      Thu Jul 19 11:12:45 [conn1055] end connection 10.18.10.210:51225
      Thu Jul 19 11:12:45 [conn1305] end connection 10.18.10.83:16212
      Thu Jul 19 11:12:45 [conn1007] end connection 10.18.10.211:39331
      Thu Jul 19 11:12:45 [conn971] end connection 10.18.10.37:54380
      Thu Jul 19 11:12:45 [conn621] end connection 10.18.10.43:45844
      Thu Jul 19 11:12:45 [conn1047] end connection 10.18.10.210:51119
      Thu Jul 19 11:12:46 [conn1323] end connection 10.18.10.83:16618
      48/187 25%
      Thu Jul 19 11:12:48 [conn925] closeAllFiles() finished
      Thu Jul 19 11:12:48 [conn925] shutdown: removing fs lock...
      Thu Jul 19 11:12:48 dbexit: really exiting now
      Logstream::get called in uninitialized state
      Thu Jul 19 11:12:49 ERROR: Client::~Client _context should be null but is not; client:conn

            Assignee:
            stephen.steneker@mongodb.com Stennie Steneker (Inactive)
            Reporter:
            stonegao Stone, Gao
            Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: