Hello everyone, We discovered Elementary recently at my company and we love the product ! I am trying to orchestrate dbt + Elementary on Databricks using workflows but I am facing some issues. Databricks has a dbt implementation that works well but you cannot run commands that don't start with dbt... As a workaround I am trying to run dbt and/or Elementary from a Python script using the subprocess library. I am facing some issues that seem to be related to os.makedirs when trying to create 'dbt_packages' or 'edr_target'. I would be happy to hear any suggestion you may have, thanks for your help and for your work on this project 😊

Hey @Samir Sayah! Thank you for reaching out, I’m really glad to hear you like and enjoy Elementary! do you have access to your

edr.log

and

dbt.log

files? If you do, can you share them? you can redact the sensitive information if there is any, and you can also share in a DM if you prefer (though for continuity, here is always best) The elementary CLI has it’s own internal dbt package which it’s using, as part of it we’re running

dbt deps

if we can’t find the packages installed in the project_dir under dbt_packages, I guess this is where you’re having the error Does the project_dir have dbt_packages where all the packages are installed at? And if not does the user running the script has the permissions to create the dir / run dbt deps? Thanks!

Hey Noa, thank you for the reply ! By trying to run 'os.mkdir("edr_target")' in my script I am getting an OSError with a code 22 : Invalid argument. I tried with both absolute and relative paths. It could be due to the file path, the absolute path of Databricks jobs looks like this :

/Workspace/Repos/.internal/ebpoe66f_commits/15pa22081d4e9a0538d35ea6e97f333fe1/data_quality

My second guess is it has to do with permissions. I will keep digging and let you know how it goes !

Sounds great! Keep us updated 😄

I was able to solve my issue by changing my working directory to something like '/dbfs/elementary/...' I am now getting a much more cryptic error: Elementary is not able to run dbt commands. I am getting many

OSError: [Errno 95] Operation not supported

Hey @Samir Sayah! Sorry for the delayed response, but I’m really glad to see you have worked everything out and all is working for you 😄 Feel free to reach out if you need anything else 👏🏻