1. do you need to perform a full-refresh in order to take advantage of the new incremental normalization? (apologies in advance if this is in the docs somewhere). i assume so given

_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh

but just want to confirm there’s no workaround as its a pretty big task 2. are the dbt logs removed?

@Chris (deprecated profile) can you help here?

i ask because

_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh

isn’t showing up anymore. and i dont see the dbt logs. so i’m guessing it is working without needing to do a full refresh?

also this is MySQL source to Snowflake destination

To populate the airbyte ab id column, yes, it needs a dbt full refresh (not an airbyte full refresh) So just recreating the normalizing tables from the airbyte raw tables instead of inserting incrementally

Dbt logs are not removed, they should be in the workspace where the sync runs from, see https://docs.airbyte.io/operator-guides/browsing-output-logs#exploring-the-logs-folders

I ask because (…) isn’t showing up anymore. and i dont see the dbt logs.

The first time Normalization 0.30.24+ runs, it will detect the missing

_airbyte_ab_id

column on normalized tables and should automatically decide to perform a dbt full refresh (same behavior as older normalization versions < 0.30.24) at least once The next time it runs, the column should now exists and populated, so it can keep on running incrementally with insert or merge statements, so you should not see that “warning” anymore is this what you are seeing?

so i’m guessing it is working without needing to do a full refresh?

(full_refresh term can be confusing with an “airbyte full_refresh” where the whole table is transferred through network again, but this is not required)

You can verify the normalization/dbt incremental behavior by: 1. does your final normalized tables include the

_airbyte_ab_id

now? (it should start getting different values for

_airbyte_normalized_at

too) 2. if you browse the workspace folder and look at the final sql files, you should see merge (or insert) statements instead of create table: https://docs.airbyte.io/operator-guides/transformation-and-normalization/transformations-with-sql#export-plain-sql-files 3. reading dbt logs

ahhh dbt full-refresh! i was thinking full-refresh = resetting the connector. that makes sense (i’m familiar with dbt incremental feature) confirming

_airbyte_ab_id

is in the tables (without resetting the connector, obviously dbt realised to perform full-refresh) also my point on dbt logs being removed. what i mean is they longer surface in the UI logs. previously we would see something like

22:21:59 | 1 of 11 OK created incremental model my_schema.my_table [SELECT in 91.02s]

but that’s not the end of the world. although would prefer to see what tables dbt creates in the UI logs

also does this feature not work for mysql --> snowflake? (reading through the PR)

also my point on dbt logs being removed. what i mean is they longer surface in the UI logs. previously we would see something like

222159 | 1 of 11 OK created incremental model my_schema.my_table [SELECT in 91.02s]

but that’s not the end of the world. although would prefer to see what tables dbt creates in the UI logs

mmm this user is also sharing airbyte sync logs where I was expecting to see the dbt logs too… That’s troublesome if they are not shown anymore This might need further investigation… https://airbytehq.slack.com/archives/C01MFR03D5W/p1636023601097400?thread_ts=1636017154.089100&amp;cid=C01MFR03D5W

also does this feature not work for mysql --> snowflake? (reading through the PR)

It should, please let me know if you get into errors there

will do. also on the logs from. a lot of different connectors the logs aren’t coming through e.g. i have a GSC connector and the logs aren’t being surfaced in the UI but when i follow the logs in cloudwatch its all there

Could you create a github ticket to address disappearing logs in the UI please?