Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-40560

Ensure system-collection protections for local and config system collections

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.3.4
    • Affects Version/s: 4.0.5
    • Component/s: Security
    • None
    • Minor Change
    • ALL
    • Hide
      original reproduction

      I'll give the exact steps I used to reproduce, including the binary that I am testing against. I'm assuming the issue is not about the binary, though.
       

      snap install juju-db
PATH="/snap/juju-db/current/bin:$PATH"
go get -v github.com/juju/testing/...
cd github.com/juju/testing
git remote add jameinel github.com/jameinel/juju-testing
git checkout mgo-with-replSet
export TEST_LOGGING_CONFIG="juju.testing=TRACE"
go test -check.v -check.f 'mgoSuite.*(Reset|Proxied)'

      It does seem to require that both TestNewProxiedSession and TestResetWhenUnauthorizied are run.

      I'm guessing it is actually an issue with the test suite tear down followed by another setup.

      I realize this isn't a minimalist reproduction with only existing mongo tools in play, but given the bit that is failing is an invariant deep in the code, it does seem like a missed expectation.

      Note also that the test does not fail 100% of the time, but probably close to 90%. So there is a bit of a race involved, just one that typically fails.

      Note the setup of Mongodb during the test suite is being done here:

      https://github.com/jameinel/juju-testing/blob/d2cd4adac5bf0330ef9cea5dcc18d485c2cb2b3c/mgo.go#L221

      And if there is no --replSet (and also no replSetInitiate call), then the tests pass 100% of the time.

      I can run other tests/try to find a smaller reproducer if that is relevant.

      Show
      original reproduction I'll give the exact steps I used to reproduce, including the binary that I am testing against. I'm assuming the issue is not about the binary, though.   snap install juju-db PATH="/snap/juju-db/current/bin:$PATH" go get -v github.com/juju/testing/... cd github.com/juju/testing git remote add jameinel github.com/jameinel/juju-testing git checkout mgo-with-replSet export TEST_LOGGING_CONFIG="juju.testing=TRACE" go test -check.v -check.f 'mgoSuite.*(Reset|Proxied)' It does seem to require that both TestNewProxiedSession and TestResetWhenUnauthorizied are run. I'm guessing it is actually an issue with the test suite tear down followed by another setup. I realize this isn't a minimalist reproduction with only existing mongo tools in play, but given the bit that is failing is an invariant deep in the code, it does seem like a missed expectation. Note also that the test does not fail 100% of the time, but probably close to 90%. So there is a bit of a race involved, just one that typically fails. Note the setup of Mongodb during the test suite is being done here: https://github.com/jameinel/juju-testing/blob/d2cd4adac5bf0330ef9cea5dcc18d485c2cb2b3c/mgo.go#L221 And if there is no --replSet (and also no replSetInitiate call), then the tests pass 100% of the time. I can run other tests/try to find a smaller reproducer if that is relevant.
    • Security 2019-05-06, Security 2019-05-20, Security 2019-06-03, Security 2019-06-17, Security 2019-07-01, Security 2019-08-12, Security 2019-08-26, Security 2019-09-09, Security 2019-09-23, Security 2019-10-07, Security 2019-10-21, Security 2019-11-04, Security 2019-12-16, Security 2019-12-30, Security 2019-01-13, Security 2019-01-27, Security 2020-02-10

      Normal write permissions are the only requirement for writing to the following collections in `local`:

      replset.election
replset.minvalid
replset.oplogTruncateAfterPoint
startup_log

      Could we require more explicit write privileges
      to prevent changing these collections?

      original description

      I'm doing testing of interacting with mongo and replicaset. And during the test suite, we start up a Mongodb, and clean it up between tests. One of our tests is testing the infrastructure can recover if a test resets a password and the server needs to be torn down and set up again.

      Without --replSet having been set, the test passes just fine. With --replSet, I get a failure with this backtrace:

       mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x5650b0d4597a]
 mongod(+0x222EC3E) [0x5650b0d44c3e]
 mongod(+0x222F112) [0x5650b0d45112]
 libpthread.so.0(+0x12890) [0x7ff109b80890]
 libc.so.6(gsignal+0xC7) [0x7ff1097bbe97]
 libc.so.6(abort+0x141) [0x7ff1097bd801]
mongod(_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j+0x0) [0x5650af3d90e4] mongod(_ZNK5mongo4repl33ReplicationConsistencyMarkersImpl11getMinValidEPNS_16OperationContextE+0x258) [0x5650af6c7d78] mongod(_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_11OplogBufferEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE+0x170) [0x5650af79c2c0]
mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_11OplogBufferEPNS0_22ReplicationCoordinatorE+0x11E) [0x5650af79d8ee]
 mongod(+0xC76EB6) [0x5650af78ceb6] mongod(_ZN5mongo8executor22ThreadPoolTaskExecutor11runCallbackESt10shared_ptrINS1_13CallbackStateEE+0x19A) [0x5650b05dc13a]
 mongod(+0x1AC682A) [0x5650b05dc82a]
 mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x1AF) [0x5650b056181f]
 mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xAA) [0x5650b0561b5a]
mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x95) [0x5650b05624c5]
 libstdc++.so.6(+0xBD57F) [0x7ff10a40057f]
 libpthread.so.0(+0x76DB) [0x7ff109b756db]
 libc.so.6(clone+0x3F) [0x7ff10989e88f]
-----  END BACKTRACE  -----
2019-04-10T17:36:10.691+0400 I NETWORK  [conn6] end connection 127.0.0.1:44610 (0 connections now open

       
      Digging through the code it seems to be this invariant:

      https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/replication_consistency_markers_impl.cpp#L117

      bool ReplicationConsistencyMarkersImpl::getInitialSyncFlag(OperationContext* opCtx) const {     
            auto doc = _getMinValidDocument(opCtx);
            invariant(doc);  // Initialized at startup so it should never be missing.

      I don't know why the doc is missing in this case.

       

       

            Assignee:
            spencer.jackson@mongodb.com Spencer Jackson
            Reporter:
            jameinel John Arbash Meinel
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: