I've found a strange behavior in one of my sharded clusters.

I will resume this report because I really don't know how to explain this, I just need to know what kind of data is needed to debug this behavior and help to fix this asap, because it is impacting some of my customers.

Basically, I have a sharded cluster UNDER PRODUCTION with 7 nodes with replication factor of 3, 6 mongos and 3 config servers.

All those nodes were created using the binary version 3.4.4 but they already passed though different versions (trying to escape of another bug): 3.4.0, 3.4.3 and 3.4.6. Currently on version 3.4.6

My shard key is composed by a hashed index on a custom string field named "pid". This field is basically one object identification but can repeat for a thousand times, for example.

My database has 500 millions documents.

All insertions in this database are performed by applications using java driver version 3.4.2. We had some old applications using java driver 3.2.2 many months ago, but this cluster was created just a month ago and I really don't believe those applications added any documents in this cluster. The above example is about one document added to the database with a java driver 3.4.2.

Explaining the problem with commands:

This find on mongo shell returns only one document:

mongos> db.investigation_cards.find({_id : ObjectId("5988e4ea8584c230ad486e43")}, {_id : 1, pid : 1})
{ "_id" : ObjectId("5988e4ea8584c230ad486e43"), "pid" : "10155585763579589_1923551154567182" }

This count, otherwise, returns two documents in mongo shell:

mongos> db.investigation_cards.count({_id : ObjectId("5988e4ea8584c230ad486e43")})
2

What is really impacting my customers is that if I execute the aggregation showed below in my application, some documents are returned twice. In one MongoDB GUI we can see the same result but in others the document doesn't returns twice (pictures aggregation_*).

db.investigation_cards.aggregate(
[
	{ "$match" : { _id : ObjectId("5988e4ea8584c230ad486e43")}},
	{ "$project" : { _id : 1, "pid" : 1}},
]
)