Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-43131

diagnostic.data and mongod.log disagree on when a connection was established

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 4.2.0
    • Component/s: Diagnostics
    • None
    • ALL
    • Hide

      git clone https://github.com/mongodb-labs/benchmarks
      cd benchmarks/sysbench/lua
      ./high_thread_count.lua run --time=720 --rate=3000 --threads=36 --idle-handlers=20 --idle-connections=400000 --csv-file=../../../sysbench.csv --percentile=99 --report-interval=1 --mongo-url='mongodb://username:password@10.2.0.200:27017/admin?replicaSet=rs0&w=3'

      For reference, my data is from this patch build: https://evergreen.mongodb.com/task/sys_perf_4.2_linux_3_node_replSet_sysbench_patch_a4b751dcf51dd249c5865812b390cfd1c0129c30_5d692e57e3c33122f4866e52_19_08_30_14_19_01#

      Show
      git clone https://github.com/mongodb-labs/benchmarks cd benchmarks/sysbench/lua ./high_thread_count.lua run --time=720 --rate=3000 --threads=36 --idle-handlers=20 --idle-connections=400000 --csv-file=../../../sysbench.csv --percentile=99 --report-interval=1 --mongo-url='mongodb://username:password@10.2.0.200:27017/admin?replicaSet=rs0&w=3' For reference, my data is from this patch build: https://evergreen.mongodb.com/task/sys_perf_4.2_linux_3_node_replSet_sysbench_patch_a4b751dcf51dd249c5865812b390cfd1c0129c30_5d692e57e3c33122f4866e52_19_08_30_14_19_01#

      I created a benchmark that opens lots of connections. I ran this with:

      ./high_thread_count.lua run --time=720 --rate=3000 --threads=36 --idle-handlers=20 --idle-connections=400000 --csv-file=../../../sysbench.csv --percentile=99 --report-interval=1 --mongo-url='mongodb://username:password@10.2.0.200:27017/admin?replicaSet=rs0&w=3'
      

      The above will open 36 connections that maintain a constant load of 2000 queries/sec and also open about 830 new idle connections per second. For keep-alive purposes the idle connections do a single query each after 300 seconds.

      In mongod.log I can see the connections opened at the start of the test:

      2019-09-02T15:36:14.137+0000 I  NETWORK  [listener] connection accepted from 10.2.0.10:50966 #82 (72 connections now open)
      2019-09-02T15:36:14.137+0000 I  NETWORK  [conn82] received client metadata from 10.2.0.10:50966 conn82: { driver: { name: "mongoc", version: "1.14.0" }, os: { type: "Linux", name: "Amazon Linux", version: "2", architecture: "x86_64" }, platform: "cfg=0x2156a0e9 posix=200809 stdc=201112 CC=GCC 7.3.1 20180303 (Red Hat 7.3.1-5) CFLAGS="" LDFLAGS=""" }
      2019-09-02T15:36:14.163+0000 I  ACCESS   [conn82] Successfully authenticated as principal username on admin from client 10.2.0.10:50966
      

      However, in the diagnostic.data only 36 connections are created at that time, and a large increase in connections is only seen 300 s later.

        1. mongod.0.tar.gz
          1.41 MB
        2. screenshot-1.png
          screenshot-1.png
          259 kB

            Assignee:
            backlog-server-platform DO NOT USE - Backlog - Platform Team
            Reporter:
            henrik.ingo@mongodb.com Henrik Ingo (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: