Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33089

Unable to start queryable mongod because it failed to regenerate index for admin.system.users

    • Fully Compatible
    • ALL
    • v3.6, v3.4
    • Platforms 2018-02-12

      Unable to start queryable mongod for the backup of mongod version 3.4.10.

      indexes on admin.system.user on mongod 3.4.10:

      db.system.users.getIndexes()
      [
      	{
      		"v" : 2,
      		"key" : {
      			"_id" : 1
      		},
      		"name" : "_id_",
      		"ns" : "admin.system.users"
      	}
      ]
      

      indexes on admin.system.user on mongod 3.4.11:

      db.system.users.getIndexes()
      [
      	{
      		"v" : 2,
      		"key" : {
      			"_id" : 1
      		},
      		"name" : "_id_",
      		"ns" : "admin.system.users"
      	},
      	{
      		"v" : 2,
      		"unique" : true,
      		"key" : {
      			"user" : 1,
      			"db" : 1
      		},
      		"name" : "user_1_db_1",
      		"ns" : "admin.system.users"
      	}
      ]
      

      Queryable restore uses the latest mongod available to start queryable mongod. So for the backup of mongod 3.4.10 it will use queryable mongod 3.4.11. While starting queryable mongod, it tries to create the extra index introduced in mongod 3.4.11, and it fails because queryable mongod is started in read only mode. Following is the error encountered:

      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] MongoDB starting : pid=20564 port=27812 dbpath=/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/ 64-bit host=brs-qr-2.vpc8.10gen.cc
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] db version v3.4.11
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] git version: 34f5bec2c9d827d71828fe858167f89a28b29a2a
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] modules: enterprise
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] build environment:
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten]     distmod: rhel62
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] options: { config: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/conf.yaml", net: { bindIp: "0.0.0.0", port: 27812, ssl: { CAFile: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/ca.pem", PEMKeyFile: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/serverIdentity.pem", mode: "requireSSL" } }, queryableBackup: { apiUri: "127.0.0.1:8095", memoryQuotaMB: 8192.0, snapshotId: "5a741effdf9db17ed7c5b844" }, security: { authorization: "enabled" }, setParameter: { authenticationMechanisms: "MONGODB-X509" }, storage: { dbPath: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/", engine: "queryable_wt", queryableBackupMode: true, wiredTiger: { engineConfig: { cacheSizeGB: 6.0 } } }, systemLog: { destination: "file", path: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/mongod.log" } }
      2018-02-02T08:19:12.258+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=6144M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),extensions=[local={entry=queryableWtFsCreate,early_load=true,config={apiUri="127.0.0.1:8095",snapshotId="5a741effdf9db17ed7c5b844",dbpath="/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath"}},],readonly=true,
      2018-02-02T08:19:18.792+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine wiredTiger when current storage engine is queryable_wt
      2018-02-02T08:19:18.955+0000 I ACCESS   [initandlisten] No authorization index detected on admin.system.users collection. Attempting to recover by creating an index with spec: { ns: "admin.system.users", v: 2, name: "user_1_db_1", key: { user: 1, db: 1 }, unique: true }
      2018-02-02T08:19:18.955+0000 F ACCESS   [initandlisten] Failed to regenerate index for admin.system.users. Exception: Cannot execute a write operation in read-only mode
      2018-02-02T08:19:18.955+0000 I STORAGE  [initandlisten] IllegalOperation: Cannot execute a write operation in read-only mode
      

            Assignee:
            spencer.jackson@mongodb.com Spencer Jackson
            Reporter:
            ankur.srivastava@mongodb.com Ankur Srivastava (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: