Hey folks blob wave, is there a CLI option for cleaning up the Elementary schema? Ours is getting a bit big so would like to have an automated method of cleaning up historical data not used in our dashboard

Hi! There is

-f true

flag when running

edr monitor

command, it cleans

alerts_v2

table, but i do not know about other. All you can do is run

edr --help

and same for subcommands and get to know if there is another CLI option you need 😃

Thanks. There doesn’t appear to be anything, I was just wondering if there was something hidden away 🙂. Looking at the schema, there doesn’t appear to be any constraints between the tables so we will probably just have to write a custom Airflow DAG that deletes from the various tables using available date columns as our filters. I guess that’s what our Beta environment is for…

Hi @Euan Blackledge have you implemented your idea with a custom dag?

Nope. We’ve got a ticket on our backlog for this. Should be easy enough to do. It’ll probably be next quarter before we tackle it

To clean it, I recreate schema using

dbt run -s elementary -t prod -f --profile elementary

Nice. If we need a clean slate, we’ll certainly use this. We’re interested in keeping the last 28 days of data so will have to use another process