-
Type:
Question
-
Resolution: Fixed
-
Priority:
Critical - P2
-
Affects Version/s: None
-
Component/s: Configuration
-
None
-
(copied to CRM)
After packaging together with the jar a ConfigProvider (the standard way external secrets are added to Kafka Connect plugins) for the purposes of getting external secrets and configuring the config provider itself (in this case a VaultConfigProvider), e.g.
"config.providers": "vault", "config.providers.vault.class": "com.github.jcustenborder.kafka.config.vault.VaultConfigProvider", "config.providers.vault.param.vault.address": "https://vault.domain.com", "config.providers.vault.param.vault.token": "atokenhere", "config.providers.vault.param.vault.login.by": "Token"
// vault host, token and path redacted
using / invoking the use of the secret in the plugin
"connection.uri": "${vault:kv/my/path:MONGO_URI}",
will result in the secret engine being created and invoked:
INFO VaultConfigProviderConfig values: vault.address = https://vault.domain.com vault.login.by = Token vault.max.retries = 5 vault.namespace = vault.prefix = vault.retry.interval.ms = 2000 vault.secret.minimum.ttl.ms = 1000 vault.ssl.verify.enabled = true vault.token = [hidden] (com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376) com.bettercloud.vault.Vault <init> INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2. INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131) INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53) INFO VaultConfigProviderConfig values: vault.address = https://vault.domain.com vault.login.by = Token vault.max.retries = 5 vault.namespace = vault.prefix = vault.retry.interval.ms = 2000 vault.secret.minimum.ttl.ms = 1000 vault.ssl.verify.enabled = true vault.token = [hidden] (com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376) Jul 22, 2022 6:32:41 AM com.bettercloud.vault.Vault <init> INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2. [2022-07-22 06:32:41,789] INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131) [2022-07-22 06:32:41,789] INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53) [2022-07-22 06:32:42,009] INFO AbstractConfig values: (org.apache.kafka.common.config.AbstractConfig:376)
but then Kafka Connect, and maybe more specifically the mongo plugin throws this error:
{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nInvalid value ${vault:my/path:MONGO_URI} for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv://\nYou can al100 1295 100 439 100 856 173 337 0:00:02 0:00:02 --:--:-- 510rType}/config/validate`"}
// vault host, token and path redacted
This has been tested both in a local docker-compose environment with confluentinc/cp-kafka-connect:7.1.0. and real AWS MSK Connect environment
It look like even though the secret engine is invoked, the plugin incorectly then uses the secret invocation/interpolation syntax itself as the value, instead of using the retrieved secret
Could you please look into this?
If this is indeed a known bug or gap it renders the plugin unusable in a production env, hence the priority
Thank you,
Florin @ LeadIQ
- is related to
-
-
- Closed
-