Hello! I was trying to ingest data from Snowflake with profiling enabled on version 0.8.44.4 and I believe it broke something during the computation time. My lineage wasn't complete, however when I don't use the SQL Profiling, I have the right lineage (I have the ignore_start_time_lineage boolean set to true for testing purposes). Do you know if I can run lineage and profiling on two separate metadata ingestion runs?

Hi @worried-zebra-47870 1. if you can share logs we can debug what broke and then fix it 🙂 2. You can definitely run lineage and profiling on two separate ingestion runs. However that may lead to some duplicate work going on in both runs (listing tables and all). If you are using stateful ingestion, I would suggest to either use separate

pipeline_name

for both or disable stateful ingestion in one of the ingestion.

I don't think I'll require stateful ingestion for profiling. Thanks for the tip!