Hi guys, Is it possible to integrate elementary wi...
# generalb
Beto Castillo
07/12/2022, 8:02 PMHi guys, Is it possible to integrate elementary with databricks?
🙌 1
e
Elon Gliksberg
07/12/2022, 8:13 PMHi Beto! Thanks for reaching us.
At the moment we do not support Databricks.
However, there's an open issue for that in our roadmap and we might implement that in the future.
👍 1
🙌 1
m
Maayan Salom
07/12/2022, 8:20 PMHey @Beto Castillo,
Adding to what @Elon Gliksberg said that this is indeed an open issue, and we are getting more demand for it recently.
We prioritize new integrations based on demand from users, and as it's coming up a lot lately I believe we will do it soon.
I'll make sure to update you as soon as we do!
🙌 2
s
Sanjay Krishna
07/12/2022, 10:35 PMOh I would love to have this. Just found about elementary today and I saw it wasn't already available.
b
Beto Castillo
07/13/2022, 12:44 PM@Maayan Salom could I use bdt test and dbt expectatios on databricks, then save the results to bigquery to be consumed by elementary?
m
Maayan Salom
07/13/2022, 12:46 PMThat's a very interesting use case! @Beto Castillo
Do you have a way to replicate tables / schema from Databricks to Bigquery?
b
Beto Castillo
07/13/2022, 1:19 PM@Maayan Salom yes, databricks have a connector to read/write data in bigQuery.
https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/big-query.html
Beto Castillo
07/13/2022, 1:21 PM@Maayan Salom to do a "replication" you have to build an etl process (batch)
m
Maayan Salom
07/13/2022, 1:24 PMCool!
So the elementary dbt package writes all the results to a schema of it's own, and the CLI reads tables from that schema.
You will need to create a profile named 'elementary' that would point the CLI to the schema in Bigquery.
The flow would be:
1. dbt package runs as part of your project (any tests you want - dbt, dbt-expectations, elementary tests)
2. results are saved to elementary tables on databricks
3. copy elementary tables to big query
4. CLI has bigquery profile
5. CLI reads the elementary tables from bigquery and generates report
Would it be easier for you to replicate a schema, or do you prefer a list of specific tables?
👍 1
Maayan Salom
07/13/2022, 1:34 PMActually @Beto Castillo, I consulted with @Or Avidov (our CTO) and he pointed out that there are some macros in the dbt-package that might not work on Databricks.. We didn't test it yet.
😔 1
Maayan Salom
07/13/2022, 1:36 PMI can find time in the next few days to test it and see if the changes are big.
Also if you/your team are up for it, we can collaborate on making the changes to the package
🙌 1
b
Beto Castillo
07/13/2022, 2:05 PMMy team would like to see a demo integrating dbt test, dbt expectations, elementary + azure databricks + bigquery. Our Client is the largest bank in Peru. Do you think that is possible?
m
Maayan Salom
07/13/2022, 3:17 PMLet's talk about it in DM ok? We can figure something out 🙂
b
Beto Castillo
07/13/2022, 3:58 PMSorry, but what is DM?
e
Elon Gliksberg
07/13/2022, 3:59 PMIt’s a Direct Message, meaning private chat.
😁 1
m
Maayan Salom
07/13/2022, 3:59 PMExactly ☺️
😆 1
b
Beto Castillo
07/13/2022, 3:59 PMgreat!
30 Views