-
Type: Improvement
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Connection Mgmt
-
None
Current connection pool implementation is based upon python set() and prone to connection pool poisoning problem.
Python set() is unordered by definition. So each command executed withdraws random connection from pool, touching its last usage time. This scheme shows good result on persistent established loads, increasing number of connections to handle the load. But spikes of loads introduces pool poisoning problem. During the spikes of load connection pool creates new connection to handle the increased load (which is expected behavior), but when returning to normal levels connection pool cannot reduce amount of connections, because due to unordered set() commands passed to virtually all open connection, preventing them from expiration. So the amount of open connections is never decreased on particular loads.
LIFO policy implementation addresses this issue. All connections are ordered by usage time using deque. Connections are taken and returned to pool from the left side of deque. For persistent load this scheme is equal to current implementation. For the spiky loads, after handling the spikes, we have frequently used connections from the left side, while unused moved to the right side, giving them chance to expire. Deque implementation offers efficient copy-free way of expiring connections from the right side.
Using proposed implementation we've observed ~18000 of mongod connection on system start, reducing to ~4000 after several minutes.
Found and implemented by https://github.com/dvolodin7 in https://github.com/mongodb/mongo-python-driver/pull/380.
- causes
-
MOTOR-377 Test failure - test.asyncio_tests.test_asyncio_client.TestAsyncIOClient.test_reconnect_in_case_connection_closed_by_mongo
- Closed
-
PYTHON-2743 Incompatibility with gevent.Timeout
- Closed
-
MOTOR-336 Test failures with PyMongo 3.8.0
- Closed
-
PYTHON-1777 Test failure - test_client.TestClient.test_max_idle_time_reaper RuntimeError: deque changed size during iteration
- Backlog