Use separate “new connection” timeout

When a user creates a new client, an explicit connection timeout context is provided (and, optionally, a separate connection timeout setting). However, when a user runs an operation and the connection pool doesn’t have any available connections, the pool will try to create a new connection using the same context as the operation (see pool.go#L457). The result is that a completely different timeout is used when creating connections for a new client as when creating connections during an operation, likely violating the user’s expectation and possibly causing a scenario where new connections are never or rarely established successfully.

Instead, we should use a separate connection timeout when creating new connections, whether when creating a new client or when executing an operation.

Establish new connections in the background

When a user runs an operation, a connection must be selected from the connection pool. If one is available, it is removed from the pool, used, then returned to the pool. If one is not available, we either wait for an available connection to be returned to the pool or create one if the connection pool is below the max size. Currently, the connection creation is attempted and cancelled synchronously with the operation, meaning if the operation times out, the connection creation must stop to maintain the expected operation timeout behavior (see pool.go#L339). In some common circumstances, creating a connection may take much longer than an individual query (e.g. when using TLS-encrypted connections to Atlas clusters and querying small documents using an index), so it may be necessary to continue creating a new connection even after the operation times out.

Instead, create new connections in a separate goroutine when necessary and wait for that or another connection to be available when running an operation. If a new operation required creating a new connection, continue creating the connection up to the default or configured connection timeout.