End client is Tomcat with webapp which uses Spring

Recently we converted a single instance mongo with ip address 1.1.1.1 (example) to a replica set with ip addresses 1.1.1.1, 2.2.2.2, 3.3.3.3 (arbiter)

Connection String : 

mongodb://username:password@1.1.1.1:27017,2.2.2.2:27017,3.3.3.3:27017/database?replicaSet=replicaset

After updating the connection string on application server to standard mongouri, the application is stalling after switchover from 1.1.1.1 to 2.2.2.2.

We enabled detailed logging for mongo driver using below logging.properties org.mongodb.driver.level=ALL 

The observation is that once the war file is loaded and during the "Cluster created with settings " is only connecting to 1.1.1.1 no matter what we do. 

The most interesting (and concerning) aspect is that we tried removing 1.1.1.1 from all the property files, removed 1.1.1.1 from the replica set (2.2.2.2 was made the PRIMARY) and shutdown mongo on 1.1.1.1 to make sure there is no way the driver could know or understand the presence of 1.1.1.1 but still as soon as I start tomcat, i could see below in the tomcat logs.

Another issue is that the application is getting stalled after switching over to 2.2.2.2 and we believe understanding and resolving this issue can bring light to the stalling issue.

Note : We have a node js app running which is connecting to this replicaset and it is able to connect to 2.2.2.2 seamlessly after a switchover from 1.1.1.1. It is only on tomcat we are facing this issue.

May 08, 2019 12:18:33 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.76
May 08, 2019 12:18:33 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
May 08, 2019 12:18:34 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/lib/javax.servlet-api-3.0.1.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
May 08, 2019 12:18:44 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster created with settings {hosts=[1.1.1.1:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log
FINE: Updating cluster description to {type=UNKNOWN, servers=[

]
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log
FINE: Closing connection connectionId{localValue:1}
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Exception in monitor thread while connecting to server 1.1.1.1:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.SocketStream.open(SocketStream.java:63)
at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57)
at com.mongodb.connection.SocketStream.open(SocketStream.java:58)
... 3 more