When a new connector configuration for the MongoDB sink connector is validated using Connect, the MongoDB sink connector includes in the validation output the `topic` property (with a value matching the `topics` property) even though no such property is defined in the ConfigDef and is not even included in the connector configuration.
For example, validate a connector configuration as follows:
curl -X PUT -H "Content-Type: application/json" --data @mongo-sink.json http://localhost:8083/connector-plugins/MongoSinkConnector/config/validate
where the mongo-sink.json file contains:
{
"name": "my-mongo-sink",
"connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
"tasks.max":1,
"connection.uri":"mongodb+srv://<user>:<password>@cluster0.abcde.mongodb.net/<dbname>?retryWrites=true&w=majority",
"database":"sink-test",
"collection":"docs",
"topics":"connect-offsets",
"batch.size":0,
"document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.KafkaMetaDataStrategy",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.storage.StringConverter"
}
Note that this configuration specified a `topics` (plural) property, but did not specify a `topic` (singular) property. Connect requires either sink connector configurations to specify the topics to be consumed by the connector via a `topics` or `topics.regex` property, but not both.
The result of the Connect REST API validation is the following JSON:
{
"name": "com.mongodb.kafka.connect.MongoSinkConnector",
"error_count": 0,
"groups": [
"Common",
"Transforms",
"Predicates",
"Error Handling",
"Connection",
"Overrides",
"Namespace",
"Writes",
"Post Processing",
"Id Strategies",
"Errors",
"Change Data Capture"
],
"configs": [
{
"definition": null,
"value": {
"name": "topic",
"value": "connect-offsets",
"recommended_values": [],
"errors": [
"Configuration is not defined: topic"
],
"visible": true
}
},
...
{
"definition": {
"name": "topics",
"type": "LIST",
"required": false,
"default_value": "",
"importance": "HIGH",
"documentation": "A list of kafka topics for the sink connector, separated by commas",
"group": "Connection",
"width": "MEDIUM",
"display_name": "The Kafka topics",
"dependents": [],
"order": 1
},
"value": {
"name": "topics",
"value": "connect-offsets",
"recommended_values": [],
"errors": [],
"visible": true
}
},
...
]
}
AK Connect is currently marking this as an error because it is not defined in the connector's ConfigDef, but that's a bug (https://issues.apache.org/jira/browse/KAFKA-10600).
Why does the MongoDB sink connector include in its validation output a `topic` configuration property when it is not required and not even supplied by the user? It's even stranger that the value of this `topic` property matches the value of the user-supplied `topics` property.
Perhaps the connector at one time looked for the `topic` property, and keeping that helps maintain backward compatibility. But if this is the only reason, then perhaps this should be transparent to users and the `MongoSinkConnector.validate(...)` method should not include `topic` in its response.
- related to
-
KAFKA-195 Sink don't validate topic config when using topics.regex
-
- Closed
-