Problem The 1st time execution of a stateful snowflake ingestion will create a checkpoint. To well support incremental ingestion, I expected the 2nd run’s lineage_start_time should be the previouls run’s execution time. Now, you have to manually set start_time, else by default it read the entire day of history query data. Describe the solution you’d like Since this is a stateful job, every execution will create a checkpoint. We can use the latest checkpoint’s execution time as the start_time. Therefore, in a scheduled ingestion, we don’t have to query 1~2 days data, but just delta data for lineage. Additional context In datahub v0.9.2.0, the data processing flow as as following. The latest checkpoint is fetched and after sink and only used for change rate calculation. Is this correct? 1. Read receipt config file for both UI and CLI. (start_time or bucket_duration is provided or using default value) 2. Run pipeline 1) Read source 2) Do transformation 3) Sink to target 4) report progress 3. Checkpoint 1) get the last checkpoint 2) compare (1 - overlapping/old) % and do the changed threshold checking 3) commit new checkpoint