For lineage, I’m trying to link the following: <database table> -> KafkaConnect Pipeline -> Kafka CDC topic. However, I’m having issues building this lineage from UI. I’m wondering if that is because pipeline lineage is only available through the API, or if maybe there’s another reason why the kafka connect pipeline doesn’t show up in search. Any help or advice in building this lineage would be appreciated! I am aware of the file-based lineage, but I’d like to avoid manually drawing lineage in a file if possible. Thanks!

This should actually work out of the box. @hundreds-photographer-13496 ^ please take a look

Hey @rapid-spoon-75609! Is the required edge not auto-generated as part of kafka connect ingestion ? - https://datahubproject.io/docs/generated/ingestion/sources/kafka-connect/

Sorry for the delayed response! Kafka Connect ingestion seems to be working, but I’m not even seeing a lineage tab in the UI:

So the lineage for populated for Kafka Connnect Pipelines is via Tasks(=DataJobs) - do you see any tasks in above pipeline. If you go to particular task -you should be able to see the lineage. Check this Integration Details doc for what to expect - https://datahubproject.io/docs/generated/ingestion/sources/kafka-connect/#integration-details

hmm, there are no tasks listed. My recipe is very basic as it’s an unauthenticated connector inside a kubernetes cluster:

source:
      type: "kafka-connect"
      config:
        # Coordinates
        connect_uri: "<http://kafka-connect-api:8083>"