Hello slack, I am trying to get elementary up and running with dbt on athena. Through cloning latest

master

branch I have the whole setup working with dbt, iceberg table creation, etc. (using

local:

and not

package:

in packages.yml) So I am very happy I got this working. Reading the PR ELE-36 which was merged into master last week (https://github.com/elementary-data/dbt-data-reliability/pull/597) it seems the

edr report

functionality should be working as well. However I can not get it to work. I have documented my steps below, including the final error I receive when running edr report. Setup dbt in own conda environment 1.

conda create -n my_dbt_athena python=3.11

2.

conda activate my_dbt_athena

3.

pip install dbt-athena-community

Initialize & test dbt project 4.

dbt init my_athena

Then configure for local situation using

aws_profile_name

5. check config with "`dbt debug`" (successful) Test basic functionality with example project 6.

dbt run

7.

dbt docs generate

8.

dbt docs serve

All with success (two tables generated, docs generated and served) So far dbt is working, let's continue with Elementary. Now I install elementary package, by cloning the latest

master

branch from git (https://github.com/elementary-data/dbt-data-reliability) and do a local install. I do this since the current release 0.12.1 does not yet include athena support, this was merged in master from this PR (https://github.com/elementary-data/dbt-data-reliability/pull/597 9.

git clone <https://github.com/elementary-data/dbt-data-reliability.git>

into a local directory: dbt-data-reliability (outside my_athena dbt project folder) 10. Edit

packages.yml

in dbt project folder to point to that local directory

packages:
  - local: /path/to/dbt-data-reliability

11. update

projects.yml

add elementary config into

projects.yml

since I can not create a new schema (my_athena_elementary), I have left the

+shema:

emtpy (which generates all the tables nicely in my original schema.

models:
  my_athena:
    # Config indicated by + and applies to all files under models/example/
    example:
      +materialized: view
  elementary:
    # left the schema empty here, since we cannot create a separate schema in athena
    +schema:

12. Install the package

dbt deps

No errors (dbt=1.7.2, dbt-labs/dbt_utils 1.1.1 also installed) 13. Instantiate the package

dbt run --select elementary

All the tables/views are created successfully. 14. Run test to validate if elementary tables are filled with data

dbt test

select * from my_schema.dbt_invocations di limit 10;

So elementary seems to be working and data observability ends up in the tables. This is good news, the issue is in the last step to get elementary-cli working and the report generated. 15. the documentation instructs to run

dbt run-operation elementary.generate_elementary_cli_profile

However this gives an error:

Adapter "athena" is not supported on Elementary.

checking the macro that triggers this error (

dbt_packages\elementary\macros\utils\cross_db_utils\generate_elementary_profile_args.sql

) shows that there is an error with the macro name:

{% macro athena__generate_elementary_cli_profile(method, elementary_database, elementary_schema) %}

which should be:

{% macro athena__generate_elementary_profile_args(method, elementary_database, elementary_schema) %}

So I renamed the macro name as shown above, and run the command:

dbt run-operation elementary.generate_elementary_cli_profile

This gives back my configuration , with some placeholder values for aws access keys and token. So I created an

elementary:

entry in

profiles.yml

which resembles the

my_athena:

entry I used for dbt (which is in the same

profiles.yml

file).

elementary:
  outputs:
    default:
      type: "athena"
      s3_staging_dir: "<my staging dir>"
      region_name: "<my region>"
      database: "awsdatacatalog"
      aws_profile_name: "<my profile name>"
      work_group: "<my workgroup>"
      schema: "<my schema>"
      threads: 1

After these changes, I pip install the elementary-data[athena] package:

pip install elementary-data[athena]

Testing if it works, with

edr --help

shows the elementary help. But it can not generate a report But when I run:

edr report

I receive the following error message (removed some confidential names):

________                          __
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
                                                 /____/

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2023-11-20 14:23:58 — INFO — Running with edr=0.12.0
2023-11-20 14:24:06 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Encountered an error while running operation: Runtime Error
  Compilation Error
    'dict object' has no attribute 'dbname'

    > in macro log_macro_results (macros\utils\log_macro_results.sql)
    > called by <Unknown>
2023-11-20 14:24:13 — INFO — Elementary's database and schema: '"<my schema name>"'
2023-11-20 14:24:13 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir <main anaoconda directory>\Anaconda3\envs\dbt_local\Lib\site-packages\elementary\monitor\dbt_project
2023-11-20 14:27:25 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error while running operation: Runtime Error
  Runtime Error
    An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: line 9:22: mismatched input ':'. Expecting: '(', '.', 'AS', '['
Please reach out to our community for help with this issue.
NoneType: None

How to fix? My gut feeling says the edr command points to the dbt_project in my conda environment :

<main anaoconda directory>\Anaconda3\envs\dbt_local\Lib\site-packages\elementary\monitor\dbt_project

and that the solution is in changing that, but I have no clue, how to do that... So any help is welcome, to point me into the right direction.. Thanks, edits were added after the initial post, to update formatting

I am this same issue for more than a month , when i ran edr send-report and same issue

2023-11-20 14:27:25 — ERROR — Could not generate the report - Error: Failed to run dbt command.

Hi @Geoffrey Stoel! Did you also install elementary CLI from master? (https://github.com/elementary-data/elementary)

@Itamar Hartstein I did pip install elementary-data[athena] So that’s through pypi I guess and I am not sure if that’s the latest master? The screen output above mentions 0.12.0 I will try a clone from master and a local pip install using pip -e and report back after.

Hi @Geoffrey Stoel, the athena support is not released yet

@Itamar Hartstein, thanks... I will try today in new clean conda environment (

my_athena_new

). Just to be sure, I will be trying the following for the edr part to get working ( 1.

git clone <https://github.com/elementary-data/elementary.git> elementary_cli_master

2.

conda activate my_athena_new

3.

pip install ./elementary_cli_master[athena]

4.

cd my_dbt_project_folder

5.

edr report

yes, sounds right

do I understand correctly that the cli uses dbt under the hood as well to get the data from the elementary schema to generate the dashboard?

yes, that's correct the CLI contains an internal dbt project that serves as the API against the database

Hurray.... 🎆 I was successful in generating my report. The issue was that the

packages.yml

file in the 'internal dbt project' for elementary (installed in my conda environment). This dbt project was pointing to an older revision of

dbt-data-reliability

. So I updated the yaml file to point to the current revision and reran

dbt deps

- git: <https://github.com/elementary-data/dbt-data-reliability.git>
    # revision: 339b4ed1ca79f4e836d1e1292319d9b98c6fccc9 
    revision: b234a2f460813d477b87ef464a7350270da0c843

However after that dbt deps run, I tried running

edr report

and I got another error: indicating there is a macro named:

can_query_relation

, which is defined in two different files (see below).

2023-11-21 10:13:51 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error:
Compilation Error
  dbt found two macros named "can_query_relation" in the project "elementary".
   To fix this error, rename or remove one of the following macros:
      - macros\utils\cross_db_utils\can_query_relation.sql
      - macros\utils\cross_db_utils\can_access_relation.sql
Please reach out to our community for help with this issue.

I compared the two and did not notice any difference so I removed one of the files. Not sure why I got two, because checking the git repo shows only the

can_query_realation.sql

Last try of

edr report

gave my my so much anticipated html report 🙂 Now time for the real evaluation and checking if elementary can play a rol in my datapipeline testing and monitoring 🙂

My issue is related to send-report command

@Ram

pip install

installs the latest official version, not what exists on master (master is a development branch)

Ok i just wanted tocheck , coz i am still having failures when running edr send-report

@Ram these are quick insructions to clone and pip install the elementary package

I am not using azure but snowflake

ah, ok... this was for (AWS) athena 😉 thought you were as well based on your response in this thread - sorry..