How does the `platform_instance_map` parameter wor...
# ingestion
n
How does the
platform_instance_map
parameter work in Kafka Connect Ingestion connector? If there are 2 different
postgres
instances each with its own
platform_instance
name such as
instance1
and
instance2
how will the map parameter look? Not sure how it would work for the same platform having multiple instances with the example - https://datahubproject.io/docs/metadata-ingestion/source_docs/kafka-connect#config-details
m
Hey @nutritious-bird-77396, that's a great question and it wouldn't be supportable currently to map (in the same recipe), two postgres platforms to different instances.. how does kafka-connect internally represent these postgres connections?
Maybe we need to build a
connection_to_platform_map
where we map the kafka-connect-specific connection name for the different postgres instances to their datahub platform and instance values.
n
As of now we are planning to address through conventions... i.e
"database.server.name"
in Kafka Connect API config matches the
platform_instance
name in the postgres ingestion. Will keep you posted on updates.
Created a PR based on this approach to prefix the
database.server.name
to the table name based on a flag - https://github.com/linkedin/datahub/pull/4321