Hi, I am trying to ingest metadata from redshift, ...
# troubleshoot
c
Hi, I am trying to ingest metadata from redshift, I wanted to understand, how the data is ingested? Like does it take change delta or scans all the tables each time the ingestion job is run. If there is any config i must use to control that, can anyone please point that out. I ask this because when i try to ingest data, datahub consumes all my cluster bandwidth and essentially chokes the redshift. This cause all other read queries to get piled up and ultimately the system crashes. If it reads all the data each time ingestion queries are run then it will be difficult to adapt datahub for our data governance purpose. Please help with this query as this is our only blocker for now. If anyone wants to connect we can discuss on direct messages as well. Thanks in Advance.
✅ 1
l
This content can't be displayed.
a
Hi @clever-magician-79463, I responded in your other thread- please copy paste links to messages in the future rather than reposting text!