Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18290

Adding a read role for a user doesn't seem to propagate to secondary until restart

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.0.3, 3.1.3
    • Affects Version/s: 3.0.2
    • Component/s: Admin, Security
    • Fully Compatible
    • ALL
    • Security 3 05/15/15

      I added the read role to the local database to workaround MMS-2240.

      For a simple 3 node replicaset, on the primary I did this:

      use admin
db.updateUser('mongo-mms', {"roles" : [ { "role" : "clusterMonitor",  "db" : "admin" }, {"role": "read", "db": "local"} ]})

centaurd:PRIMARY> db.system.users.find()
...
{ "_id" : "admin.mongo-mms", "user" : "mongo-mms", "db" : "admin", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "QujDz6Fl67kimiVH9WohLg==", "storedKey" : "BtDWckJVHLOF6Y/LaeUMuzRBI/0=", "serverKey" : "0GNKkjRRKddCUrIY+/wL0o2C4aY=" } }, "roles" : [ { "role" : "clusterMonitor", "db" : "admin" }, { "role" : "read", "db" : "local" } ] }
...

      Almost immediately the problem was resolved on the primary:

      2015-05-01T11:41:07.349+0100 I ACCESS   [conn28532] Successfully authenticated as principal mongo-mms on admin
2015-05-01T11:41:16.064+0100 I NETWORK  [conn28530] end connection 10.220.54.121:38399 (10 connections now open)
2015-05-01T11:41:16.064+0100 I NETWORK  [initandlisten] connection accepted from 10.220.54.121:38470 #28533 (12 connections now o
pen)
2015-05-01T11:41:16.072+0100 I NETWORK  [conn28531] end connection 10.220.54.122:36979 (10 connections now open)
2015-05-01T11:41:16.076+0100 I NETWORK  [initandlisten] connection accepted from 10.220.54.122:37024 #28534 (11 connections now o
pen)
2015-05-01T11:41:16.083+0100 I ACCESS   [conn28533] Successfully authenticated as principal __system on local
2015-05-01T11:41:16.094+0100 I ACCESS   [conn28534] Successfully authenticated as principal __system on local
2015-05-01T11:41:22.077+0100 I QUERY    [conn28437] assertion 13 not authorized for query on local.oplog.rs ns:local.oplog.rs que
ry:{ $query: {}, $orderby: { $natural: 1 } }
2015-05-01T11:41:22.077+0100 I QUERY    [conn28437]  ntoskip:0 ntoreturn:-1
2015-05-01T11:41:46.097+0100 I NETWORK  [conn28533] end connection 10.220.54.121:38470 (10 connections now open)
2015-05-01T11:41:46.097+0100 I NETWORK  [initandlisten] connection accepted from 10.220.54.121:38542 #28535 (12 connections now o
pen)
2015-05-01T11:41:46.106+0100 I NETWORK  [conn28534] end connection 10.220.54.122:37024 (10 connections now open)
2015-05-01T11:41:46.109+0100 I NETWORK  [initandlisten] connection accepted from 10.220.54.122:37065 #28536 (12 connections now o
pen)
2015-05-01T11:41:46.115+0100 I ACCESS   [conn28535] Successfully authenticated as principal __system on local
2015-05-01T11:41:46.127+0100 I ACCESS   [conn28536] Successfully authenticated as principal __system on local
2015-05-01T11:42:16.133+0100 I NETWORK  [conn28535] end connection 10.220.54.121:38542 (10 connections now open)
2015-05-01T11:42:16.136+0100 I NETWORK  [conn28536] end connection 10.220.54.122:37065 (9 connections now open)

      However the secondary logs continue to contain the not-authorized warnings for more than one hour after:

      2015-05-01T13:10:17.059+0100 I ACCESS   [conn27021] Successfully authenticated as principal __system on local
2015-05-01T13:10:27.653+0100 I QUERY    [conn26950] assertion 13 not authorized for query on local.oplog.rs ns:local.oplog.rs query:{ $query: {}, $orderby: { $natural: 1 } }
2015-05-01T13:10:27.653+0100 I QUERY    [conn26950]  ntoskip:0 ntoreturn:-1
2015-05-01T13:10:46.746+0100 I NETWORK  [conn27020] end connection 10.220.54.122:41619 (5 connections now open)
2015-05-01T13:10:46.747+0100 I NETWORK  [initandlisten] connection accepted from 10.220.54.122:41659 #27022 (6 connections now open)

      I verified that the secondary had replicated the creds:

      * mongo centaur-2.dev.man.com
centaurd:SECONDARY> db.setSlaveOk(true)
centaurd:SECONDARY> db.system.users.find()
{ "_id" : "admin.mongo-mms", "user" : "mongo-mms", "db" : "admin", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "QujDz6Fl67kimiVH9WohLg==", "storedKey" : "BtDWckJVHLOF6Y/LaeUMuzRBI/0=", "serverKey" : "0GNKkjRRKddCUrIY+/wL0o2C4aY=" } }, "roles" : [ { "role" : "clusterMonitor", "db" : "admin" }, { "role" : "read", "db" : "local" } ] }

      Restarting the secondary seems to fix the issue.

      The issue is also visible in MMS in the centaurd cluster - the MMS agent starts detecting the oplog window for the primary immediately, but the secondary doesn't have this metric until restart. Screenshot attached.

        1. oplog-window.png
          8 kB
          James Blackburn

            Assignee:
            spencer.jackson@mongodb.com Spencer Jackson
            Reporter:
            jblackburn James Blackburn
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: