hi, I have an ingestion troubleshooting question. ...
# ingestion
b
hi, I have an ingestion troubleshooting question. I have tried to execute code similar to this example: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/examples/library/lineage_emitter_kafka.py and I don't get any errors, but also do not see the lineage connection show up in the UI if I run the same code but change the sink to the REST connector it works. my question is how I might go about debugging this? I looked in the containers at the kafka topics/messages and gms logs and didn't see anything that indicated an error or failure, but maybe I"m not looking in the right place note: I am running the "quickstart" DataHub on an EC2 instance. I've been able to run ingestion recipes that use the kafka sink to pull datasets in
l
@boundless-room-44377 is your single node DataHub quickstart installation using your company's Kafka infrastructure or is it using the local Kafka container? is the lineage emitter writing to the same Kafka or is it writing to a different Kafka instance?
We need to ensure DataHub is able to read the topic that the emitter is writing to
b
yes, good questions
it is using the kafka container on the instance (local)
aaand... I found the issue now
the code I used was missing the
emitter.flush()
step 🤦
but now I know how to tail. all the logs I need 🙂
l
👍