Summary
When using the MongoClient on Kubernetes to access a MongoDb database I get a segmentation fault where something in the library appears to be attempting to access/overwrite protected memory.
Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).
I've tested against versions 2.21.0 and 2.19.0
The MongoDB topology is a replicaset with two nodes (one primary and one secondary)
How to Reproduce
Create a .NET Console application and deploy it to Kubernetes with 4 Gi of ephemeral storage
In Program.cs add the following code:
var mongoUrl = new MongoUrlBuilder() { Username = <USERNAME>, UseTls = true, ReplicaSetName = <RS>, AuthenticationMechanism = "GSSAPI", Servers = new List<MongoServerAddress> { new MongoServerAddress("S1", PORT), new MongoServerAddress("S2", PORT) } }; using var loggerFactory = LoggerFactory.Create(b => { b.ClearProviders(); b.AddSimpleConsole(); b.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Trace); }); var settings = MongoClientSettings.FromConnectionString(mongoUrl.ToMongoUrl().ToString()); Console.WriteLine($"mongourl {mongoUrl.ToMongoUrl().ToString()}"); settings.LoggingSettings = new LoggingSettings(loggerFactory); MongoClient dbClient = new MongoClient(settings);Console.WriteLine("made it to db client creation"); try { dbClient.StartSession(); Console.WriteLine("getting database"); var database = dbClient.GetDatabase("DBNAME"); Console.WriteLine("running command"); var test = database.RunCommand<BsonDocument>(new BsonDocument("ping", 1)); Console.WriteLine($"end - running command {test}");} catch (Exception ex) { Console.WriteLine(ex.ToString()); } Console.WriteLine("Pinged your deployment. You successfully connected to MongoDB!");
Additional Background
Via logging statements I have confirmed that the connection string is formatted as expected by the URL builder, I was able to get the MongoDB client trace logs using the logging configuration above.. hopefully that'll be helpful to your debugging:
mongourl mongodb://username@Secondary_Server:Port,Primary_Server:Port/?authMechanism=GSSAPI;tls=true;replicaSet=RS
dbug: MongoDB.SDAM[0]
1 Initialized
dbug: MongoDB.SDAM[0]
1 Cluster opening
dbug: MongoDB.SDAM[0]
1 Secondary_Server Port Adding server
dbug: MongoDB.SDAM[0]
1 Secondary_Server Port Added server
dbug: MongoDB.SDAM[0]
1 Primary_Server Port Adding server
dbug: MongoDB.SDAM[0]
1 Primary_Server Port Added server
dbug: MongoDB.SDAM[0]
1 Description changed
dbug: MongoDB.SDAM[0]
1 Secondary_Server Port Server opening
dbug: MongoDB.Connection[0]
1 Secondary_Server Port Connection pool opening 600000 0 100 2 120000 500
dbug: MongoDB.Connection[0]
1 Secondary_Server Port Connection pool created 600000 0 100 2 120000 500
dbug: MongoDB.Internal.IServerMonitor[0]
1 Secondary_Server Port Initializing
dbug: MongoDB.Internal.IServerMonitor[0]
1 Secondary_Server Port Initialized
dbug: MongoDB.SDAM[0]
1 Secondary_Server Port Server opened
dbug: MongoDB.SDAM[0]
1 Primary_Server Port Server opening
dbug: MongoDB.Connection[0]
1 Primary_Server Port Connection pool opening 600000 0 100 2 120000 500
dbug: MongoDB.Connection[0]
1 Primary_Server Port Connection pool created 600000 0 100 2 120000 500
dbug: MongoDB.Internal.IServerMonitor[0]
1 Primary_Server Port Initializing
dbug: MongoDB.Internal.IServerMonitor[0]
1 Primary_Server Port Initialized
dbug: MongoDB.SDAM[0]
1 Primary_Server Port Server opened
dbug: MongoDB.SDAM[0]
1 Cluster opened
dbug: MongoDB.Internal.RoundTripTimeMonitor[0]
1 Primary_Server Port Monitoring started
dbug: MongoDB.Internal.RoundTripTimeMonitor[0]
1 Secondary_Server Port Monitoring started
made it to db client creation
starting session
dbug: MongoDB.ServerSelection[0]
1 CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector
, OperationsCountServerSelector } (null) (null) { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Secondary_Server:Port" }", EndPoint: "Unspecified/Secondary_Server:Port", ReasonChanged: "ServerInitialDescription", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", LastHeartbeatTimestamp: null, LastUpdateTimestamp: "2023-09-18T20:42:12.6463312Z" }, { ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Primary_Server:Port" }", EndPoint: "Unspecified/Primary_Server:Port", ReasonChanged: "ServerInitialDescription", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", LastHeartbeatTimestamp: null, LastUpdateTimestamp: "2023-09-18T20:42:12.6488517Z" }] } Server selection started
info: MongoDB.ServerSelection[0]
1 CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector
, OperationsCountServerSelector } (null) (null) { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Secondary_Server:Port" }", EndPoint: "Unspecified/Secondary_Server:Port", ReasonChanged: "ServerInitialDescription", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", LastHeartbeatTimestamp: null, LastUpdateTimestamp: "2023-09-18T20:42:12.6463312Z" }, { ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Primary_Server:Port" }", EndPoint: "Unspecified/Primary_Server:Port", ReasonChanged: "ServerInitialDescription", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", LastHeartbeatTimestamp: null, LastUpdateTimestamp: "2023-09-18T20:42:12.6488517Z" }] } Waiting for suitable server to become available 29917
dbug: MongoDB.Connection[0]
1 Secondary_Server Port Connection pool ready
dbug: MongoDB.SDAM[0]
1 Secondary_Server Port Description changed { ServerId: "
", EndPoint: "Unspecified/Secondary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff952b1c549aaa3945f506"), "counter" : NumberLong(4) }, Type: "ReplicaSetSecondary", WireVersionRange: "[0, 13]", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9204932Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9204940Z" }
dbug: MongoDB.SDAM[0]
1 Description changed
dbug: MongoDB.SDAM[0]
1 3 Secondary_Server Port Heartbeat started
dbug: MongoDB.ServerSelection[0]
1 CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector
, OperationsCountServerSelector } (null) (null) { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Secondary_Server:Port" }", EndPoint: "Unspecified/Secondary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff952b1c549aaa3945f506"), "counter" : NumberLong(4) }, Type: "ReplicaSetSecondary", WireVersionRange: "[0, 13]", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9204932Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9204940Z" }, { ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Primary_Server:Port" }", EndPoint: "Unspecified/Primary_Server:Port", ReasonChanged: "ServerInitialDescription", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", LastHeartbeatTimestamp: null, LastUpdateTimestamp: "2023-09-18T20:42:12.6488517Z" }] } Server selection succeeded Secondary_Server Port
dbug: MongoDB.Connection[0]
1 Primary_Server Port Connection pool ready
dbug: MongoDB.SDAM[0]
1 Primary_Server Port Description changed { ServerId: "
", EndPoint: "Unspecified/Primary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff93a049fd067ff988e8ff"), "counter" : NumberLong(7) }, Type: "ReplicaSetPrimary", WireVersionRange: "[0, 13]", ElectionId: "7fffffff0000000000000008", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9386605Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9386609Z" }
getting database
running command
dbug: MongoDB.SDAM[0]
Initializing (maxSetVersion, maxElectionId): Saving tuple (setVersion, electionId) of (6, 7fffffff0000000000000008) as (maxSetVersion, maxElectionId) for replica set "dev-112823-eieio-rs0" because replica set primary Unspecified/Primary_Server:Port sent (6, 7fffffff0000000000000008), the first (setVersion, electionId) tuple ever seen for replica set "dev-112823-eieio-rs0".
dbug: MongoDB.SDAM[0]
1 Description changed
dbug: MongoDB.SDAM[0]
1 4 Primary_Server Port Heartbeat started
dbug: MongoDB.ServerSelection[0]
1 CompositeServerSelector{ Selectors = WritableServerSelector, LatencyLimitingServerSelector
, OperationsCountServerSelector } (null) create { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Secondary_Server:Port" }", EndPoint: "Unspecified/Secondary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff952b1c549aaa3945f506"), "counter" : NumberLong(4) }, Type: "ReplicaSetSecondary", WireVersionRange: "[0, 13]", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9204932Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9204940Z" }, { ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Primary_Server:Port" }", EndPoint: "Unspecified/Primary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff93a049fd067ff988e8ff"), "counter" : NumberLong(7) }, Type: "ReplicaSetPrimary", WireVersionRange: "[0, 13]", ElectionId: "7fffffff0000000000000008", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9386605Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9386609Z" }] } Server selection started
dbug: MongoDB.ServerSelection[0]
1 CompositeServerSelector{ Selectors = WritableServerSelector, LatencyLimitingServerSelector
, OperationsCountServerSelector } (null) create { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Secondary_Server:Port" }", EndPoint: "Unspecified/Secondary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff952b1c549aaa3945f506"), "counter" : NumberLong(4) }, Type: "ReplicaSetSecondary", WireVersionRange: "[0, 13]", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9204932Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9204940Z" }, { ServerId: "
{ ClusterId : 1, EndPoint : "Unspecified/Primary_Server:Port" }", EndPoint: "Unspecified/Primary_Server:Port", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("64ff93a049fd067ff988e8ff"), "counter" : NumberLong(7) }, Type: "ReplicaSetPrimary", WireVersionRange: "[0, 13]", ElectionId: "7fffffff0000000000000008", LastHeartbeatTimestamp: "2023-09-18T20:42:12.9386605Z", LastUpdateTimestamp: "2023-09-18T20:42:12.9386609Z" }] } Server selection succeeded Primary_Server Port
dbug: MongoDB.Connection[0]
1 Primary_Server Port Connection checkout started
dbug: MongoDB.Connection[0]
1 Primary_Server Port Connection adding
dbug: MongoDB.Connection[0]
1 5 Primary_Server Port Connection created
dbug: MongoDB.Connection[0]
1 5 Primary_Server Port Connection opening
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sending
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sent
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Receiving
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Received
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sending
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sent
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Receiving
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Received
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sending
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sent
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Receiving
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Received
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sending
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Sent
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Receiving
trce: MongoDB.Connection[0]
1 5 Primary_Server Port Received
malloc_consolidate(): unaligned fastbin chunk detected
/bin/sh: line 1: 7 Aborted (core dumped) dotnet project_name.dll
please note:
due to company policy servers, ports, and identifying information cannot be in logs hence primary_server, secondary_server etc
We had similar code working for a previous replicaset that was torn down, I believe there is a connectivity or authentication issue but the error message is getting written to someplace it shouldn't be in the memory, which is halting troubleshooting. Any help would be appreciated!