4 (Data Node) + 1 (Arbiter)

 

4 Data nodes are setup in different region and data centers. 

The arbiter is setup in the same region as one of the data node but in a different data center. That specific data node spams the MongoDB log with:

{"t":\{"$date":"2024-09-11T06:00:49.256+01:00"}

,"s":"I",  "c":"REPL",     "id":3xxxx07, "ctx":"ReplCoordExtern-4","msg":"Cannot select sync source because it is not readable","attr":{"syncSourceCandidate":"xxxxxxxx:2xxxx"}}

{"t":\{"$date":"2024-09-11T06:00:49.257+01:00"}

,"s":"I",  "c":"REPL",     "id":3xxxx07, "ctx":"ReplCoordExtern-4","msg":"Cannot select sync source because it is not readable","attr":{"syncSourceCandidate":"xxxxxxx:2xxxx"}}

This cause the log file to be around 22GB per day. 

23G Sep  4 06:00 mongodb.log.2024-09-04T05-00-48
24G Sep  5 06:00 mongodb.log.2024-09-05T05-00-50
24G Sep  6 06:00 mongodb.log.2024-09-06T05-00-49
22G Sep  7 06:00 mongodb.log.2024-09-07T05-00-48
23G Sep  8 06:00 mongodb.log.2024-09-08T05-00-47
25G Sep  9 06:00 mongodb.log.2024-09-09T05-00-47
26G Sep 10 06:00 mongodb.log.2024-09-10T05-00-47
25G Sep 11 06:00 mongodb.log.2024-09-11T05-00-49

It seems that its trying to select a sync source that is an arbiter.