Hey all, I'm trying to make Elementary working with dbt-athena. I know that there is no official support yet, but there are some working branches from

artem-garmash

What it works: • I'm able to install elementary dbt packages using dbt deps ✅ • I'm able to bootstrap elementary tables/views ✅ • When I run a model tables in my elementary schema are populated ✅ Now, the I have issues when I try to run this

edr report

(installed via

pip install git+<https://github.com/artem-garmash/elementary@athena-support-v3>

I know I know, not ufficially supported yet). I configured of course my profiles for elementary, manually. I get this error:

2023-10-20 14:34:32 — INFO — Running with edr=0.11.0
2023-10-20 14:34:37 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Encountered an error:
Parsing Error
  at path ['incremental_strategy']: Undefined is not valid under any of the given schemas
2023-10-20 14:34:47 — ERROR — Failed to parse Elementary's database and schema.
2023-10-20 14:34:47 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/vscode/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir . --target development_staging
2023-10-20 14:34:52 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error:
Parsing Error
  at path ['incremental_strategy']: Undefined is not valid under any of the given schemas
Please reach out to our community for help with this issue.

it somehow looks for

incremental_strategy

and I'm wondering where do I have to define it...I'm clueless...

For @Nicola Corda or anyone trying the branches, the branch will not work if default

table_type

will be set to

iceberg

. Without that, it should work using mix of iceberg tables for incremental tables and hive tables for the rest (e.g. temp tables), including edr report and alerts

Perfect, many thanks 💯

Hi @Nicola Corda! Thanks a lot for looking into this! Let me know if you'd like to have a call to discuss it further / if we can help with anything.

I believe that we can setup a call with @Artem Garmash too, what he did is great work, I tested yesterday and today 💯 I was able to plug elementary for a PoC in my enviroment and (for now) generate reports.

@Nicola Corda / @Itamar Hartstein / @Artem Garmash great work you’re doing on this. I tried a fresh install from the master branch today and I have everything working (had to rename 1 macro), until the report part. Not sure how to fix that honestly, so posted a question with all my steps / details in it. https://elementary-community.slack.com/archives/C02CTC89LAX/p1700509560573959 Does this ring a bell with you?

@Geoffrey Stoel @Itamar Hartstein was was the issue then? If I understood correctly seems due to the fact that elementary was not realesed yet

looks like the released version used first. And after installing from master, conflicts with the previously installed package. @Nicola Corda, i think you had similar problem

my issue was that I was using the the wrong development branch,

athena-support-v3

instead of

athena-support-v3-ci

my issue was familiar with @Nicola Corda issue - no branch configured (or my awareness at that moment 😉 ).. I was running the elementary-cli through pip install (so picking the PyPi released version) and not latest commit. By switching pulling from latest master branch, pip installing from local and making sure the packages.yml for the internal dbt project the elementary package \was also pointing to the latest commit for the dbt-data-reliability and run dbt deps on it). a little bit more detailed steps/explanation in the thread under my original post: https://elementary-community.slack.com/archives/C02CTC89LAX/p1700509560573959

but after all is released that should work in a bit easier way I guess, just pointing elementary cli from specific pypi version and then using in dbt a pointer to that release right?

I assume that when both are included in a release, no specific pointers are necessary anymore, because the cli package will also use the release version of the dbt-data-reliability package. But I am pretty new to this whole thing and don't know how the release cycle/process for both packages are managed from a dependency perspective.

Yup I aspect the same. • cli ->

elementary-data 0.13.0

via pypi • in dbt deps ->

packages:
  - package: elementary-data/elementary
    version: 0.13.0

for example

sorry about the delayed response yes after both are released there will be no need for any unusual configuration! I'll update when it's live

@Itamar Hartstein athena support is live now in 0.12.1 right? https://github.com/elementary-data/elementary/releases/tag/v0.12.1 looking at release notes seems in.

Hey @Nicola Corda! yes the release is out! We'll send an announcement in #C02BPB78J4T soon.