Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-93

Connector issue with high load: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to failure to restore tailable cursor position

    • Type: Icon: Task Task
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.1
    • Component/s: None
    • None
    • Environment:
      MongoDB: 3.6.17, Kafka Connector 1.1

      Hi, I am using MongoDB: 3.6.17 in my setup. since MongoDB version < 4.0 does not support change streams in database level. I am registering multiple connector instances on my setup. I have 10 connector instances listening to 10 collections in my setup. I try to load around 5000 documents in the collections. After some time I start seeing failures like. and the connector fails and the related Kafka topic does not only partial messages.

      Also when I look into the connector status in curl -X GET http://localhost:8083/connectors/mongo-source-2/status the result says its running. but the connector is in a failed state.

       

      2020-03-30 17:00:15,513] WARN [Producer clientId=confluent.monitoring.interceptor.connector-producer-mongo-source-7-0] Error while fetching metadata with correlation id 3 : {_confluent-monitoring=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
      com.mongodb.MongoQueryException: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to failure to restore tailable cursor position. Last seen record id: RecordId(6810047002907247540)"' on server mongo1:27017
      com.mongodb.MongoQueryException: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(6810046998612279328)"' on server mongo1:27017
      com.mongodb.MongoQueryException: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to failure to restore tailable cursor position. Last seen record id: RecordId(6810047020087116556)"' on server mongo1:27017
      com.mongodb.MongoQueryException: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(6810046998612281250)"' on server mongo1:27017
      com.mongodb.MongoQueryException: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(6810047015792149212)"' on server mongo1:27017
      com.mongodb.MongoQueryException: Query failed with error code 136 and error message 'Error in $cursor stage :: caused by :: errmsg: "CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(6810047020087116658)"' on server mongo1:27017

        1. connector-config.json
          0.5 kB
          Sabari Gandhi

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            sabari.mgn@gmail.com Sabari Gandhi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: