Hey everyone,
Can we have our dbt target be say, d...
# community-supportj
jordan fox
04/29/2025, 9:42 PMHey everyone,
Can we have our dbt target be say, databricks, and have our elementary target go somewhere else like azure sql? How does that work?
n
Noya Offer
04/30/2025, 2:05 PMHey @jordan fox!
Unfortunately this is not possible at the moment, as dbt can only work against one DB connection at a time and the Elementary schema is part of the same pipeline.
The best option to achieve this result would probably be another sync job from Databricks to Azure SQL.
j
jordan fox
04/30/2025, 5:07 PMI think the issue is that the on-end-hook takes a good 2-3 mins to complete when say a 13s refresh operation happened. Is there any way to address this? We're about to switch to dagster where it'll run each table individually and that's going to be a lot of compute just updating the elementary artifacts.
jordan fox
04/30/2025, 5:09 PMWe were hoping that maybe elementary would be quicker writing to an oltp instance rather than the roundabout way its writing into databricks.
jordan fox
05/04/2025, 3:42 AMJust got databricks running with dagster and the various runs all targeting individual assets. Elementary is taking 2-4 times as long to capture the metadata in the post hook than the actual runs taking 5-15 seconds to materialize the assets. Going to need to work on a workaround here.
jordan fox
05/04/2025, 3:43 AMwondering if it'd be viable to intercept insert_rows, replace_table_data, delete_and_insert and instead have it asynchronously send to an azure function or something and handle the data from there.
jordan fox
05/04/2025, 3:43 AMare there other functions to intercept?
n
Noya Offer
05/04/2025, 12:22 PMI understand!
As for the
on-run-enddbt_project.yml Copy code
vars:
columns_upload_strategy: nonedbt_columns7 Views