Dear Elementary community, I want to check if source freshness anomality checks are applicable in our project. We ingest data daily, but databases contain only latest data (i.e. no historical snapshots) . So we don't have freshness history for training anomality checks. The question: does elementary record freshness information to some table? Is it possible to configure freshness anomality checks in our case?

Hey @Denis Kokorin, Unfortunately the package's tests will not be able to be trained without continous historical data (at least for the training period) I can say that we're currently working on tests that are based on external updates to tables. But that's going to be presented first on our cloud product, due to technical limitations

Thanks @Daniel Pollak, could you clarify what's the main difference between freshness anomality and row count anomality? AFAIK row count anomality doesn't require continuous historical data and stores row count into elementary metrics table. So theoretically if freshness results are also stored in metrics table - it should be possible, right?

@Denis Kokorin, it isn't different between them, or any other anomaly test in elementary. more information about the different behaviors is specified in our docs if anything is unclear, let me know drop corrupt happy