Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-46902

Single-server replica set responds about 3 times slower than a standalone server on the same hardware

    • Type: Icon: Bug Bug
    • Resolution: Community Answered
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Fully Compatible
    • ALL
    • Hide

      I cannot share the application code that demonstrates the 3 times difference and running Mongo Shell script doesn't show as much of a difference. Not sure if you will be able to reproduce the same.

      Anyway, what I did is this:

      On Windows, add and remove these two lines in mongod.cfg between tests:

      replication:
        replSetName: rs0 

      Each time after changing these lines, I ran the application test or the attached test script (not in the same server run).

      The test script attempts to mimic the application calls, but for much smaller documents and for a real application test find commands are running against a different collection.

      Show
      I cannot share the application code that demonstrates the 3 times difference and running Mongo Shell script doesn't show as much of a difference. Not sure if you will be able to reproduce the same. Anyway, what I did is this: On Windows, add and remove these two lines in mongod.cfg between tests: replication: replSetName: rs0 Each time after changing these lines, I ran the application test or the attached test script (not in the same server run). The test script attempts to mimic the application calls, but for much smaller documents and for a real application test find commands are running against a different collection.

      This might be a Mongo C driver issue, not a database issue, but since the only variable changed between my tests is the server configuration, I created this under SERVER.

      I am running Mongo DB v4.2.1 on Windows and the Mongo C driver v1.16.1-1.el8 on CentOS 8. This is a development machine, so the Mongo DB server is configured to run as a replica set with a single server. CentOS is running in HyperV, but looking at the CPU usage in VTune profiling results below, this is not caused by overloading CPUs.

      What I noticed is that when I run an application test against this server configured as a standalone server, the test runs about 3 times faster than when it is configured as a replica set.

      I tried to reproduce the same problem with a small Mongo Shell script and while it doesn't show the same difference, the test against a replica set still runs about 1.3-1.5 times slower against a replica set (~32s vs. ~48s). Sometimes the difference is as little as 1.2 times, but never as close as on Windows. The same test running Mongo Shell on Windows differs only about 1.1 times and is more consistent in terms of variance.

      The two VTune screenshots below are bottom-up stacks showing the same application test doing about 80K look-ups by _id and single-document inserts in a database that contains about 2M documents. Same database in both tests. The only change is adding and removing replSetName in the configuration file and restarting the database server.

      This VTune screenshot is against a standalone server. The bottom two green lines show CPU usage for application 2 threads interacting with Mongo C. You can see how each is using about 50-60% of CPU and the test runs for 90 seconds. The wait time in the poll function isĀ 61.8 seconds for both threads combined.

      This VTune screenshot shows the exact same application test running against the exact same Mongo DB server, but configured as a replica set. Notice on the CPU usage scale that the duration of the test is now 260 seconds and CPU usage for each application thread is barely 25%. The wait time now is 418 seconds for both threads combined.

      The call stack is exactly the same for both screenshots.

      Looking at Mongo C driver source, I see thatĀ _mongoc_socket_wait uses a fixed timeout, which is consistent with VTune profiling results, but I'm not sure if it's the server that changes send/receive timing or it's the client that polls differently. I didn't have a chance to profile the source.

      Let me know if you want me to try anything in my configuration.

        1. mongodb-200k-ins-only-rs-sa-perfmon.png
          mongodb-200k-ins-only-rs-sa-perfmon.png
          217 kB
        2. mongodb-80k-find-ins-rs.png
          mongodb-80k-find-ins-rs.png
          326 kB
        3. mongodb-80k-find-ins-sa.png
          mongodb-80k-find-ins-sa.png
          326 kB
        4. mongod-excessive-journal-flush.png
          mongod-excessive-journal-flush.png
          264 kB
        5. mongo-rs-vs-sa-insert.js
          0.7 kB

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            cis74633@bell.net Andre M
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: