-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Unknown
-
Affects Version/s: 1.6.1
-
Component/s: Source
-
Needed
-
Add copy.existing.allow.disk.use configuration.
Allow the copy existing aggregation to use temporary disk storage if required. Defaults to true but can be disabled if the user doesn't have the permissions
for disk access.
============
Was: Exceeded memory limit for $group, but didn't allow external sort
Hi Team,
The Mongo Src connector is failing with the error :
connect | org.apache.kafka.connect.errors.ConnectException: com.mongodb.MongoCommandException: Command failed with error 16945 (Location16945): 'Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.' on server inclusio-1-shard-00-02.kwe2e.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "ok": 0.0, "errmsg": "Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.", "code": 16945, "codeName": "Location16945", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "signature": {"hash": {"$binary": {"base64": "TZhmmTYxktZ6ivXdEWZkImFgqVA=", "subType": "00"}}, "keyId": 6991155560347336706}}} connect | Caused by: com.mongodb.MongoCommandException: Command failed with error 16945 (Location16945): 'Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.' on server inclusio-1-shard-00-02.kwe2e.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "ok": 0.0, "errmsg": "Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.", "code": 16945, "codeName": "Location16945", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "signature": {"hash": {"$binary": {"base64": "TZhmmTYxktZ6ivXdEWZkImFgqVA=", "subType": "00"}}, "keyId": 6991155560347336706}}}
The Src connector properties are fairly simple
name = prodMongoSrc connector.class = com.mongodb.kafka.connect.MongoSourceConnector tasks.max = 1 errors.retry.timeout = -1 errors.retry.delay.max.ms = 10000 errors.log.enable = true errors.log.include.messages = true connection.uri = mongodb+srv:/<######> topic.prefix = prod copy.existing = true offset.partition.name = partition-03
I have hashed the credentials.
When i relaunch it works but again fails after sometimes.
Looking for support and help on this