Hey wave Since the latest upgrade we have an on run end hook Elementary Community #community-support

Hey :wave: Since the latest upgrade we have an on...

Josh Elston-Green

12/04/2023, 5:14 PM

Hey 👋 Since the latest upgrade we have an on-run-end hook executing a query despite having these vars set:

Copy code

# Elementary vars stopping artefact upload on dev schemas
  disable_run_results: "{{ (target.name not in ['prod','circleci_prod']) | as_bool }}"
  disable_tests_results: "{{ (target.name not in ['prod','circleci_prod']) | as_bool }}"
  disable_dbt_artifacts_autoupload: "{{ (target.name not in ['prod','circleci_prod']) | as_bool }}"
  disable_dbt_invocation_autoupload: "{{ (target.name not in ['prod','circleci_prod']) | as_bool }}"

The query being executed in dev is the

Copy code

INSERT INTO data_monitoring_metrics

Is there a new var I need to add or is this a bug with this particular model?

Samuel Garfield

12/04/2023, 8:59 PM

Same over here. What's more, when elementary runs in our dbt cloud CI job it is attempting to write to the production

data_monitoring_metrics

table instead of the CI env table

Noa Kurman

12/05/2023, 9:22 AM

Hey @Josh Elston-Green! Did you upgrade to 13.0.0?

Samuel Garfield

12/06/2023, 9:39 PM

@Noa Kurman can't speak for Josh but I am encountering this error while using

Copy code

- package: elementary-data/elementary
    version: 0.13.0

Samuel Garfield

12/06/2023, 9:47 PM

I've tried a couple different ways of doing this, but none of them work. I just tried running a CI job with this in our

vars

Copy code

vars:
  disable_dbt_artifacts_autoupload: "{{ target.name == 'ci' }}"
  disable_tests_results: "{{ target.name == 'ci' }}"
  disable_run_results: "{{ target.name == 'ci' }}"
  disable_dbt_invocation_autoupload: "{{ target.name == 'ci' }}"

Despite this on-run-end hooks are still running in our CI environment (where the job's target is set to

ci

). I confirmed

{{ target.name == 'ci' }}

evaluates to

True

by adding a model named

test.sql

with these contents:

select {{ target.name == 'ci' }}

When I download the compiled code from the dbt Cloud artifacts, the contents are:

select True

Samuel Garfield

12/07/2023, 5:57 PM

also for further context this is on

dbt=1.6.7

and

databricks=1.6.6

Maayan Salom

12/10/2023, 3:12 PM

Hey @Samuel Garfield , I @Josh Elston-Green answered here, could you check if this answers your use case as well? https://elementary-community.slack.com/archives/C02CTC89LAX/p1702220625648309?thread_ts=1702031013.575389&cid=C02CTC89LAX

Samuel Garfield

12/14/2023, 9:39 PM

@Maayan Salom this doesn't really answer my question. I want to enable elementary anomaly tests in CI, but not artifact, invocation, test result, or run result uploads

Samuel Garfield

12/14/2023, 9:39 PM

is this possible?

Ofek Weiss

01/08/2024, 7:51 AM

Hi, inserts to

data_monitoring_metrics

are a part of the anomaly tests - they cannot be disabled if you with to use elementary anomaly tests drop corrupt 02

garfieldthumbsup 1

Samuel Garfield

01/08/2024, 8:32 PM

@Ofek Weiss that's true, but in principle anomaly tests only need to write to the tables specifically related to anomaly tests, and therefore one could run anomaly tests while avoiding other types of upload (eg. dbt artifacts and invocations), right? if so, are you saying that there's no way to just just disable the types of post-run hooks that aren't required to run anomaly detection tests?

Ofek Weiss

01/09/2024, 8:26 AM

@Samuel Garfield it is possible (the way you did it), but this does not completely disable the

on-run-end

hook, it will still write to the tables specifically related to anomaly tests. you will still see the post-run-end hook running, as its all running in the same hook, but the dbt artifacts will not be uploaded

garfieldthumbsup 1

32 Views

Open in Slack

Previous Next