Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21660

serverStatus can stall and trigger timeout error on SCCC distributed lock with v3.0 mongos and v3.2 cluster

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.0-rc3
    • Component/s: Sharding
    • None
    • Fully Compatible
    • ALL
    • Sharding E (01/08/16), Sharding F (01/29/16), Sharding 10 (02/19/16)

      Sample error from the log:

      sh21263| 2015-11-24T10:57:56.518-0500 F -        [thread-0] terminate() called. An exception is active; attempting to gather more information
      sh21263| 2015-11-24T10:57:56.518-0500 I NETWORK  [LockPinger] SyncClusterConnection connecting to [ren-desktop:21260]
      sh21263| 2015-11-24T10:57:56.518-0500 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
      sh21263| 2015-11-24T10:57:56.518-0500 D NETWORK  [LockPinger] connected to server ren-desktop:21260 (127.0.1.1)
      sh21263| 2015-11-24T10:57:56.518-0500 I NETWORK  [LockPinger] SyncClusterConnection connecting to [ren-desktop:21261]
      sh21263| 2015-11-24T10:57:56.518-0500 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
      sh21263| 2015-11-24T10:57:56.518-0500 F -        [thread-0] DBException::toString(): 13648 server ren-desktop:21260 in cluster ren-desktop:21260 did not respond within max network delay of 266ms
      sh21263| Actual exception type: mongo::TimeNotFoundException
      sh21263|
      sh21263|  0x14cfb7b 0x14cf402 0x7fede6356306 0x7fede6356351 0x7fede63ae271 0x7fede5bc7f8e 0x7fede58f1a0d
      sh21263| ----- BEGIN BACKTRACE -----
      sh21263| {"backtrace":[{"b":"400000","o":"10CFB7B"},{"b":"400000","o":"10CF402"},{"b":"7FEDE62F8000","o":"5E306"},{"b":"7FEDE62F8000","o":"5E351"},{"b":"7FEDE62F8000","o":"B6271"},{"b":"7FEDE5BC0000","o":"7F8E"},{"b":"7FEDE57F7000","o":"FAA0D"}],"processInfo":{ "mongodbVersion" : "3.2.0-rc3-85-gd954a34", "gitVersion" : "d954a344c157ecefd3a30b0902f2201ba77e54a0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.8.0-35-generic", "version" : "#50-Ubuntu SMP Tue Dec 3 01:24:59 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF9B28F000", "elfType" : 3, "buildId" : "C34731DD03A6781F7B90E4B441D073D3B4C33F65" }, { "b" : "7FEDE6807000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "8143139EDE058AD7AA1DA2E9972D3843F712A758" }, { "b" : "7FEDE6603000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "30D12A669C22B52949902AB59B41738C4790E149" }, { "b" : "7FEDE62F8000", "path" : "/usr/local/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FEDE5FF3000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DEB41EC41A35E9184C4679C4E5AFF6E3D06B4348" }, { "b" : "7FEDE5DDD000", "path" : "/usr/local/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FEDE5BC0000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "23D6B805A9970B893E3C17D8756A69A26CE99DB9" }, { "b" : "7FEDE57F7000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30990788EB9EC583D2006CDE9A7841E43B2884FB" }, { "b" : "7FEDE6A0F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "B93C43C481B8B1680F449F697B8FD92695F545CE" } ] }}
      sh21263|  mongod(_ZN5mongo15printStackTraceERSo+0x2B) [0x14cfb7b]
      sh21263|  mongod(+0x10CF402) [0x14cf402]
      sh21263|  libstdc++.so.6(+0x5E306) [0x7fede6356306]
      sh21263|  libstdc++.so.6(+0x5E351) [0x7fede6356351]
      sh21263|  libstdc++.so.6(+0xB6271) [0x7fede63ae271]
      sh21263|  libpthread.so.0(+0x7F8E) [0x7fede5bc7f8e]
      sh21263|  libc.so.6(clone+0x6D) [0x7fede58f1a0d]
      sh21263| -----  END BACKTRACE  -----
      

      the exception is being thrown from here:
      https://github.com/mongodb/mongo/blob/r3.0.7/src/mongo/s/distlock.cpp#L466-l471

        1. filter
          417 kB
          Randolph Tan

            Assignee:
            randolph@mongodb.com Randolph Tan
            Reporter:
            randolph@mongodb.com Randolph Tan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: