Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26654

ExceededTimeLimit: Operation timed out after uprgade from 3.2.8 to 3.2.10

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.10
    • Component/s: Networking, Sharding
    • None
    • ALL
    • Sharding 2016-11-21

      We are running our production cluster using version 3.2.8, Our cluster consists of four mongos , three config servers and ten shards-3-node-replica-sets , we perform both primary and secondary reads. Very frequently we hit SERVER-26159 and our mongos crashes. According to the bug report the issue was solved on 3.2.10 and we upgraded on Oct 11. 10 minutes after the upgrade completed we start receiving errors that the mongos can't connect to the shards due to timeout. Timeouts were very frequent and reduce our processing throughput to 20%. We forced to rollback to 3.2.8 and the issue gone away, we don't receive any timeout but the SERVER-26159 returned. 

      Oct 11 06:40:03 [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to node1-48668:48668
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to node0-49179:49179
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node0-49627:49627
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node0-49630:49630 - ExceededTimeLimit: Operation timed out
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node1-49631:49631
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node0-48668:48668
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node0-48574:48574
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node0-49631:49631 - ExceededTimeLimit: Operation timed out
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node0-49631:49631 - ExceededTimeLimit: Operation timed out
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node1-48574:48574 - ExceededTimeLimit: Operation timed out

      Is this a known bug in 3.2.10 and is there any workaround?

        1. log-20161012-app0.gz
          5.37 MB
          Antonis Giannopoulos
        2. log-20161012-node0-49631.gz
          2.90 MB
          Antonis Giannopoulos
        3. sysbench-config
          4 kB
          Antonis Giannopoulos

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            antogiann Antonis Giannopoulos
            Votes:
            2 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: