Major - P3 After upgrading to 3.4. backup for a deployment stopped working. Investigation into the mongod logs shows following error while trying to apply oplogs.
m.xgen.svc.brs.util.GenericMongoManager$MongoManagerStartException: Upgrade command failed. Cmd: `[numactl, --interleave=all, /opt/mongodb/mongodb-linux-x86_64-rhel62-3.4.0/bin/mongod, --bind_ip=0.0.0.0, --noauth, --slowms=1000, --setParameter, diagnosticDataCollectionEnabled=false, --dbpath=/data/backups/daemon/510357167fe227e9f1832923/rs0/head/, --setParameter, ttlMonitorEnabled=false, --setParameter, failIndexKeyTooLong=false, --port=27503, --upgrade, --storageEngine=mmapv1, --nojournal, --setParameter, internalValidateFeaturesAsMaster=false]` Code: `134`
StdOut:
2016-11-29T03:22:17.130+0000 I CONTROL [initandlisten] MongoDB starting : pid=22877 port=27503 dbpath=/data/backups/daemon/510357167fe227e9f1832923/rs0/head/ 64-bit host=brs27.nj1.10gen.cc
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] db version v3.4.0
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] modules: none
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] build environment:
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] distmod: rhel62
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] distarch: x86_64
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-11-29T03:22:17.131+0000 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 27503 }, operationProfiling: { slowOpThresholdMs: 1000 }, security: { authorization: "disabled" }, setParameter: { diagnosticDataCollectionEnabled: "false", failIndexKeyTooLong: "false", internalValidateFeaturesAsMaster: "false", ttlMonitorEnabled: "false" }, storage: { dbPath: "/data/backups/daemon/510357167fe227e9f1832923/rs0/head/", engine: "mmapv1", journal: { enabled: false } }, upgrade: true }
2016-11-29T03:22:17.295+0000 I - [initandlisten] Fatal assertion 18523 InvalidOptions: The field 'create' is not a valid collection option. Options: { capped: true, create: "oauthnonces", flags: 1, size: 1048576 } at src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 876
2016-11-29T03:22:17.296+0000 I - [initandlisten]
***aborting after fassert() failure
The version for the first backup snapshot is 3.0.7 (2015). The deployment dates back to 2013 (when collection oauthnonces was first created).
Find the `oauthnonces` dataset attached.