-
Type: Bug
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 2.2.0
-
Component/s: Admin
-
None
-
Environment:ubuntu on EC2
-
ALL
Was running a repair with the following command and got this assertion error in the log of 1 out of 3 shards. The other shards had a different assertion error, which I filed a different bug for:
ubuntu@mongo3:~$ mongod --version
db version v2.2.0, pdfile version 4.5
Fri Jan 18 20:56:42 git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
sudo -u mongodb mongod --dbpath=/db/mongodb --repair --repairpath=/export/mongodb --nojournal
Fri Jan 18 03:25:59 [initandlisten] 47127700/861940368 5%
Fri Jan 18 03:26:09 [initandlisten] 47232800/861940368 5%
Fri Jan 18 03:26:19 [initandlisten] 47342500/861940368 5%
Fri Jan 18 03:26:29 [initandlisten] 47451000/861940368 5%
Fri Jan 18 03:26:39 [initandlisten] 47560200/861940368 5%
Fri Jan 18 03:26:49 [initandlisten] 47665300/861940368 5%
Fri Jan 18 03:26:59 [initandlisten] 47774400/861940368 5%
Fri Jan 18 03:27:09 [initandlisten] 47881800/861940368 5%
Fri Jan 18 03:27:19 [initandlisten] 47988500/861940368 5%
Fri Jan 18 03:27:29 [initandlisten] 48097900/861940368 5%
Fri Jan 18 03:27:39 [initandlisten] 48207400/861940368 5%
Fri Jan 18 03:27:49 [initandlisten] 48313200/861940368 5%
Fri Jan 18 03:27:59 [initandlisten] 48422400/861940368 5%
Fri Jan 18 03:28:00 [initandlisten] pb3 Assertion failure la <= 0xffffff src/mongo/db/btree.h 243
0xade6e1 0x803dfd 0x683ebb 0xb1e868 0xae7d19 0xae15ed 0xae46e7 0x747ed3 0x7490bd 0xae2c0d 0x737f9c 0x8535bc 0x8560c9 0x856269 0x74037a 0x5596d8 0x55b78d 0x55c6f0 0x55d2dd 0x563dbe
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x803dfd]
mongod(ZN5mongo12BucketBasicsINS_12BtreeData_V1EE9_pushBackENS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingES3+0x8ab) [0x683ebb]
mongod(_ZN5mongo12BtreeBuilderINS_12BtreeData_V1EE6addKeyERNS_7BSONObjENS_7DiskLocE+0x2a8) [0xb1e868]
mongod(_ZN5mongo24buildBottomUpPhases2And3INS_12BtreeData_V1EEEvbRNS_12IndexDetailsERNS_21BSONObjExternalSorterEbRSt3setINS_7DiskLocESt4lessIS7_ESaIS7_EEPNS_5CurOpEPNS_12SortPhaseOneERNS_19ProgressMeterHolderERNS_5TimerE+0x1f9) [0xae7d19]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xc9d) [0xae15ed]
mongod(_ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib+0x217) [0xae46e7]
mongod(_ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE+0x1f3) [0x747ed3]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x93d) [0x7490bd]
mongod(_ZN5mongo17ensureHaveIdIndexEPKc+0x7ad) [0xae2c0d]
mongod(_ZN5mongo21ensureIdIndexForNewNsEPKc+0x9c) [0x737f9c]
mongod(_ZN5mongo6Cloner2goEPKcRKNS_12CloneOptionsERSt3setISsSt4lessISsESaISsEERSsPi+0xd1c) [0x8535bc]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x139) [0x8560c9]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) [0x856269]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x52a) [0x74037a]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x5596d8]
mongod() [0x55b78d]
mongod(_ZN5mongo14_initAndListenEi+0x480) [0x55c6f0]
mongod(_ZN5mongo13initAndListenEi+0x1d) [0x55d2dd]
mongod() [0x563dbe]
Fri Jan 18 03:28:00 [initandlisten] Assertion: 16394:reading doc for external sort failed:errno:2 No such file or directory
0xade6e1 0x8036eb 0x80386c 0x909535 0x909c10 0xae7bff 0xae15ed 0xae46e7 0x747ed3 0x7490bd 0xae2c0d 0x737f9c 0x8535bc 0x8560c9 0x856269 0x74037a 0x5596d8 0x55b78d 0x55c6f0 0x55d2dd
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0x8036eb]
mongod() [0x80386c]
mongod(_ZN5mongo21BSONObjExternalSorter12FileIterator4nextEv+0xc5) [0x909535]
mongod(_ZN5mongo21BSONObjExternalSorter8Iterator4nextEv+0x220) [0x909c10]
mongod(_ZN5mongo24buildBottomUpPhases2And3INS_12BtreeData_V1EEEvbRNS_12IndexDetailsERNS_21BSONObjExternalSorterEbRSt3setINS_7DiskLocESt4lessIS7_ESaIS7_EEPNS_5CurOpEPNS_12SortPhaseOneERNS_19ProgressMeterHolderERNS_5TimerE+0xdf) [0xae7bff]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xc9d) [0xae15ed]
mongod(_ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib+0x217) [0xae46e7]
mongod(_ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE+0x1f3) [0x747ed3]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x93d) [0x7490bd]
mongod(_ZN5mongo17ensureHaveIdIndexEPKc+0x7ad) [0xae2c0d]
mongod(_ZN5mongo21ensureIdIndexForNewNsEPKc+0x9c) [0x737f9c]
mongod(_ZN5mongo6Cloner2goEPKcRKNS_12CloneOptionsERSt3setISsSt4lessISsESaISsEERSsPi+0xd1c) [0x8535bc]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x139) [0x8560c9]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) [0x856269]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x52a) [0x74037a]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x5596d8]
mongod() [0x55b78d]
mongod(_ZN5mongo14_initAndListenEi+0x480) [0x55c6f0]
mongod(_ZN5mongo13initAndListenEi+0x1d) [0x55d2dd]
Fri Jan 18 03:28:08 [initandlisten] exception in initAndListen: 16394 reading doc for external sort failed:errno:2 No such file or directory, terminating
Fri Jan 18 03:28:08 dbexit:
Fri Jan 18 03:28:08 [initandlisten] shutdown: going to close listening sockets...
Fri Jan 18 03:28:08 [initandlisten] shutdown: going to flush diaglog...
Fri Jan 18 03:28:08 [initandlisten] shutdown: going to close sockets...
Fri Jan 18 03:28:08 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jan 18 03:28:08 [initandlisten] shutdown: closing all files...
Fri Jan 18 03:28:10 [initandlisten] 119/824 14%
Fri Jan 18 03:28:12 [initandlisten] 402/824 48%
Fri Jan 18 03:28:14 [initandlisten] 448/824 54%
Fri Jan 18 03:28:16 [initandlisten] 455/824 55%
Fri Jan 18 03:28:18 [initandlisten] 464/824 56%
Fri Jan 18 03:28:20 [initandlisten] 573/824 69%
Fri Jan 18 03:28:22 [initandlisten] 580/824 70%
Fri Jan 18 03:28:24 [initandlisten] 605/824 73%
Fri Jan 18 03:28:26 [initandlisten] 671/824 81%
Fri Jan 18 03:28:27 [initandlisten] closeAllFiles() finished
Fri Jan 18 03:28:27 [initandlisten] shutdown: removing fs lock...
Fri Jan 18 03:28:27 dbexit: really exiting now
- related to
-
SERVER-8241 assertion failures (saveerrmsg) during repair
- Closed
-
SERVER-8917 buildBottomUpPhases2And3 should check assertion code when detecting a dup
- Closed