Hello everyone, I've been attempting to install Elementary in our data warehouse for the past two months. DBT rebuilds models when changes have occurred from a previous manifest to a new manifest. I believe Elementary touches every table with post hooks and streams, although I'm not entirely certain whether the streams trigger rebuilds, but the post hooks certainly do. As a result, our pipeline, re-building all models, takes approximately 6 hours to build and another 2 hours to run tests, totalling 8 hours. Due to this extended runtime, it's challenging to merge changes within a window where our data sources won't become misaligned, leading to valid but unintended test failures (this is what's made merging it onto our main branch near impossible). My request is whether it's possible to install Elementary only on a subset of schemas in Snowflake? and if so could additional schemas be added further down the line. I can't see this mentioned anywhere in the documentation.

Hi @Josh McCarter, I do not believe it is currently impossible to enable elementary only on a subset of schemas, but I'm not sure i understand your usecase exactly. Care to elaborate more about how your data pipeline process looks, where it runs? how it runs (which commands)? how exactly do you check for changes between manifests? and any other info you are willing to share would be greatly appreciated 🙏

Our pipelines run in gitlab DWH is in snowflake. We use the dbt states to re-build models that have been modified (I think we use the catch all state vs say body) - https://docs.getdbt.com/reference/node-selection/methods#the-state-method

dbt run -m state:modified+

Then all of the tests for the re-built models are ran afterwards. Our models are separated into schemas - in and out of fashion but we could take advantage of it if theres a way to only install elementary onto given schemas vs all schemas.

Hmm i see, tho im not sure why would elementary cause changes to all of the models in the manifest. can you share the exact steps of your pipeline 🙏 ? (a DM would also be great if you prefer), id like to try an replicate it and see what problems elementary causes and why