In the Kafka Connect Ingestion its not clear to me what

provided_configs

property for? https://datahubproject.io/docs/metadata-ingestion/source_docs/kafka-connect I am having an issue where the kafka connect ingestion also ingests the destination topics but it ingests with

PROD

as env instead of the provided

env

value in the recipe such as

STG

just wondering if the provided_configs could be a solution here.

Hey, @nutritious-bird-77396, you can set env field in recipe for datahub 0.8.25 onwards to STG

source:
  type: "kafka-connect"
  config:
    #other configs
    env : STG

Thanks @hundreds-photographer-13496 for your response! I already have the

env

config specified in my recipe. Even then when connecting the lineage to the kafka topic it connects to the

PROD

env one that it created. Is there another parameter to specify the env for the topic that's created?

source:
  type: "kafka-connect"
  config:
    env: "STG"
    connect_uri: "<http://kafka-connect.svc.cluster.local:8083>"
    cluster_name: "connect-cluster"

sink:
  type: "datahub-rest"
  config:
    server: "<http://datahub-gms-service.svc.cluster.local:8080>"

oh, yes, I was just going through the code for kafka connect connector and seems like self.config.env is not passed when creating lineage dataset urns, hence default PROD is being used. https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/kafka_connect.py#L888-L905

Perfect this is exactly what i was looking for. Thanks! I will test and create a PR.