Uploaded image for project: 'Ruby Driver'
  1. Ruby Driver
  2. RUBY-1241

Mongo ruby driver caches unreachable replica ipaddresses

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 2.5.0
    • Affects Version/s: 2.4.3
    • Component/s: Replica Set
    • None
    • Environment:
      kubernetes
    • 5

      TL;DR : mongodb ruby driver tries to communicate with cluster members using their ip address which may become invalid over time, as the cluster topology evolves; after a while, existing clients cannot communicate with the cluster anymore ; caching the host name instead of the volatile ip address would fix the issue.

      I am running a mongodb replicaSet as kubernetes statefulset. In this setup, each member of cluster member runs in a "pod" (a docker container); each time I recreate the "pod", its host name (and DNS host) remains stable but the pod ip address changes.

      In other words, a cluster of 3 replicas may be formed of:

      • pod1 (hostname = mongodb-0.mycluster, ip=10.128.30.39) # <-- arbitrary address in 10.128.x.x
      • pod2 (hostname = mongodb-1.mycluster, ip=10.129.31.50) # <-- arbitrary address in 10.129.x.x
      • pod3 (hostname = mongodb-2.mycluster, ip=10.130.50.19) # <-- arbitrary address in 10.130.x.x

      If I delete the first pod, kubernetes will recreate it and the cluster will look like this:

      • pod1 (hostname = mongodb-0.mycluster, ip=10.128.10.79) # <--- same hostname, different ipaddress
      • pod2 (hostname = mongodb-1.mycluster, ip=10.129.31.50) # <--- previous hostname & ipaddress
      • pod3 (hostname = mongodb-2.mycluster, ip=10.130.50.19) # <--- previous hostname & ipaddress

      Let's explore with in Ruby console:

      puts ENV['MONGODB_URL']
      # => mongodb://myapp-api:ba8c827ae12b33ae1eab@mongodb.backends:27017/db2?replicaSet=rs0
      client = Mongo::Client.new ENV['MONGODB_URL']
      client.cluster.servers.each do |s|
        puts "#{s.address} => #{s.address.instance_variable_get(:@resolver).host}"
      end
      # mongodb-2.mongodb.backends.svc.cluster.local:27017 => 10.131.0.14 # <-- volatile ip
      # mongodb-1.mongodb.backends.svc.cluster.local:27017 => 10.130.0.15 # <-- volatile ip
      # mongodb-0.mongodb.backends.svc.cluster.local:27017 => 10.129.0.19 # <-- volatile ip
      # now, let's use the client
      client.collections.count
      # => 8
      # lets' do the same while deleting the pod containing the mongodb master
      client.collections.count
      # => Mongo::Error::SocketError: end of file reached
      # =>  	from (irb):28
      # lets' do one more time -- at this point we should have a new master
      client.collections.count
      # => Mongo::Error::SocketTimeoutError: execution expired
      # 	from (irb):29
      # let's enable the debugging output
      Mongoid.logger.level = 0
      # after a bit, these start showing
      D, [2017-09-04T16:15:07.530841 #1] DEBUG -- : MONGODB | execution expired
      D, [2017-09-04T16:15:20.537943 #1] DEBUG -- : MONGODB | No route to host - connect(2) for 10.131.0.14:27017
      D, [2017-09-04T16:15:33.545593 #1] DEBUG -- : MONGODB | No route to host - connect(2) for 10.131.0.14:27017
      D, [2017-09-04T16:15:46.554009 #1] DEBUG -- : MONGODB | No route to host - connect(2) for 10.131.0.14:27017
      D, [2017-09-04T16:15:59.561493 #1] DEBUG -- : MONGODB | No route to host - connect(2) for 10.131.0.14:27017
      
      

      Where 10.131.0.14:27017 is the former ip address of the cluster member that we just created.

      My problem is that the Ruby mongo driver caches the ip address of the cluster members upon the first connection, then it never update these addresses. After recreating each pod once, none of the original ip addresses are valid anymore, and the mongodb client is unable to process any query anymore.

      I traced it to the `Mongo::Address` class memoizing a `@resolver` instance variable, which embeds a socket created with the host ip instead of the host name. See https://github.com/mongodb/mongo-ruby-driver/blob/master/lib/mongo/address.rb#L188 .

      I have the following monkeypatch as workaround, which instanciates sockets passing the stable host name as argument instead of the volatile ip address.

      require 'mongo/address'
      
      module Mongo
        class Address
          
      
          def initialize_resolver!(ssl_options)
            return Unix.new(seed.downcase) if seed.downcase =~ Unix::MATCH
      
            family = (host == LOCALHOST) ? ::Socket::AF_INET : ::Socket::AF_UNSPEC
            error = nil
            ::Socket.getaddrinfo(host, nil, family, ::Socket::SOCK_STREAM).each do |info|
              begin
                #res = FAMILY_MAP[info[4]].new(info[3], port, host)
                # >>>>>>
                # the monkeypatch forces mongodb to use hostname for its sockets
                # instead of caching a volatile ipaddress which will break when
                # the kubernetes pods are recreated.
                res = FAMILY_MAP[info[4]].new(host, port, host)
                # <<<<<<
                res.socket(connect_timeout, ssl_options).connect!(connect_timeout).close
                return res
              rescue IOError, SystemCallError, Error::SocketTimeoutError, Error::SocketError => e
                error = e
              end
            end
            raise error
          end
      
        end
      end
      

            Assignee:
            emily.stolfo Emily Stolfo
            Reporter:
            Pervillé Julien [X]
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: