-
Type:
Bug
-
Resolution: Works as Designed
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.6.17, 4.0.18
-
Component/s: None
-
None
-
ALL
We are running a mongodb v3.6 replSet for several months (or even years) already without any problems so far. However, recently our database client, which is using mongodb java driver v3.11 for a decent time already, got the following error:
ERROR 2020-06-07 18:31:11,705 [ttp-nio-8080-exec-13] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.mongodb.MongoQueryException: Query failed with error code 261 and error message 'cannot add session into the cache' on server mongo-opensearch-03.db00.pro06.eu.idealo.com:27017] with root cause [DirectJDKLog.java:175] com.mongodb.MongoQueryException: Query failed with error code 261 and error message 'cannot add session into the cache' on server mongo-opensearch-03.db00.pro06.eu.idealo.com:27017 at com.mongodb.operation.FindOperation$1.call(FindOperation.java:735)
It turned out that the default of 1 million logical sessions had been reached. We even could not kill any sesssion:
opensearch:PRIMARY> db.runCommand( { killAllSessions: [ ] } )
{
"ok" : 0,
"errmsg" : "cannot add session into the cache",
"code" : 261,
"codeName" : "TooManyLogicalSessions"
}
We had to restart mongod in order to reset all logical sessions.
opensearch:SECONDARY> db.aggregate( [ { $listLocalSessions: { allUsers: true } },{$group:{_id:"$user",count:{$sum:1}}} ] )
{ "_id" : "admin@admin", "count" : 1 }
{ "_id" : "mms@admin", "count" : 3 }
We then upgraded from mongodb v3.6 to 4.0.18 to see if this solves the issue. It did not. After 12 days, the number of logical sessions is still constantly increasing:
opensearch:SECONDARY> db.aggregate( [ { $listLocalSessions: { allUsers: true } },{$group:{_id:"$user",count:{$sum:1}}} ] )
{ "_id" : "opensearch@admin", "count" : 115 }
{ "_id" : "admin@admin", "count" : 2388 }
{ "_id" : "mms@admin", "count" : 1370 }
I'll attach also a screenshot of our dashboard where you see that the number of logical sessions is constantly increasing.
The user "opensearch" is our usual database client. The number of logical sessions of 125 is reasonably low.
The user "mms" is mongodb's CloudManager agent. It's been updated recently to the newest version. However, the number of logical sessions of 1370 is quite high already.
Although we have been running dozens of mongodb v3.6 replSets in production for several years now, this has never happened before. Since everything is rolled out automatically, and therefore uses identical configuration, we cannot explain the error and therefore suspect a mongodb bug.
