Some notes regarding dbt-elementary. I just thought it may be interesting for community. I'm running now every dbt model as separate airflow task using K8sPodOperator. In such setup elementary on-run-start and on-run-end hooks take 1-2 minutes. Let's say it's 1,5 minutes average. For the project with many models (~350) and for hourly processing schedule interval it becomes a problem. Especially for simple models with usual run duration up to 1 minute. So possible solution may be to group models somehow and run them grouped. However we may face some obstacles in this case: • we'll lose model-level lineage in airflow • logging in airflow will become strange, so need to find a way how to work with rendered queries in case of failures • retries in dbt may be a proble (I did not try this feature yet, so it's time to check it out). I suppose that setting specific

DBT_TARGET_PATH

for every models group may help in this case Will appreciate any thoughts, suggestions regarding optimal elementary usage on big dbt projects with ~low latency required.

Hi @Serhii Dimchenko! the main reason for running each model as a separate task is the lineage in Airflow? Another option you can consider is disabling everything that runs during the on-run-end except the part that uploads the model result and schedule the rest to run separately. More context - On the

on-run-end

hooks Elementary extracts data from the dbt

results

and

metadata

objects, and runs SQL queries to load this data to the Elementary models. There are 2 types of models that Elementary updates : 1. Metadata and project objects and configurations - Such as

dbt_models

,

dbt_tests

,

dbt_sources

. 2. Result models - Such as

dbt_run_results

,

elementary_test_results

,

dbt_invocations

. 1. So you can disable the auto uploading of the metadata models (results have to be uploaded during the on-run-end). 2. Make sure to upload the metadata models yourself anytime you make a change to the project (merge a PR). How? 1. Add this to dbt_project.yml:

vars:
  disable_dbt_artifacts_autoupload: true

2. Make sure to run

dbt run --select edr.dbt_artifacts

upon merging PRs. Also you can configure this to lower the number of columns metadata that are being uploaded (there is also a flag to disable it) -

vars:
  upload_only_columns_with_descriptions: true

oh, thanks for your reply. However I will not get test results and model results in this case ( I suppose it's better to combine models into single airflow task and use

dbt retry

feature to handle failures

yeah I think grouping is the right approach but if you only disable the metadata upload and not the results upload - you will be able to get and see the model and test results

ah, now I've got your idea: upload metadata on PR event only and run results on every dbt invocation. Sounds interesting and this may that what I need actually

Yeah exactly

many thanks for such advice ! 💯