when I do dbt build all the elementary tables e g dbt invoca Elementary Community #community-support

when I do dbt_build, all the elementary tables (e....

Kam

10/28/2023, 3:31 PM

when I do dbt_build, all the elementary tables (e.g. dbt_invocations) got drop by "create or replace"...ending up losing all the data. when I do dbt_build --select <model_name>, everything work as expected. Anyone has any idea of why 'dbt build' will cause all the elementary tables to be recreated? running elementary version 0.12.0

👀 1

✅ 1

Erik Zaadi

10/29/2023, 11:01 AM

Hey there @Kam,

dbt build

should be added to

dbt_invocations

and not dropped and recreated. What version of dbt and

dbt-data-reliability

package are you using?

Kam

10/29/2023, 1:47 PM

dbt 1.3 ... will be upgrade to 1.6 next week

Kam

10/29/2023, 1:50 PM

I use - package: elementary-data/elementary version: 0.12.0

Erik Zaadi

10/29/2023, 1:56 PM

We highly recommend working with dbt 1.4 and up, see here for the explanation of why they are recreated each run.

Erik Zaadi

10/29/2023, 2:02 PM

@Kam Could you update here after you upgrade to 1.6? Would love to hear that it worked out for you

Kam

10/29/2023, 2:27 PM

oh...so, the issue is my dbt version ... Thanks a lot

✅ 1

🙏 1

Kam

10/29/2023, 2:54 PM

I figure out the issue and fixed it but don't understand why it is not working right initially ... in my dbt_project.yml my models default is

materialized = table

when I look at

dbt_invocations.sql

it shows that it is

materialized = 'incremental',

that should override my default setting of

'table'

Doesn't seem like it is being override, so when I move the default

materialized

from

models

to my project, it fixed the issue. BEFORE

Copy code

models:
  enabled: true 
  materialized: table
  transient: false

  elementary:
    +database: "dig_{{env_var('DBT_SNOWFLAKE_ENV')}}_common_db"
    +schema: "elementary"

  dig_snowflake_disw:

AFTER

Copy code

models:
  elementary:
    +database: "dig_{{env_var('DBT_SNOWFLAKE_ENV')}}_common_db"
    +schema: "elementary"

  dig_snowflake_disw:
    enabled: true 
    materialized: table
    transient: false

dbt_invocations.sql

Copy code

{{
  config(
    materialized = 'incremental',
    transient=False,
    unique_key = 'invocation_id',
    on_schema_change = 'append_new_columns',
    full_refresh=elementary.get_config_var('elementary_full_refresh'),
    meta={
      "timestamp_column": "created_at",
      "prev_timestamp_column": "generated_at",
      }
  )
}}

{{ elementary.get_dbt_invocations_empty_table_query() }}

30 Views

Open in Slack

Previous Next