gunu
11/04/2021, 1:56 AM_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh
but just want to confirm there’s no workaround as its a pretty big task
2. are the dbt logs removed?user
11/04/2021, 3:12 AMuser
11/04/2021, 6:30 AM_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh
isn’t showing up anymore. and i dont see the dbt logs. so i’m guessing it is working without needing to do a full refresh?gunu
11/04/2021, 6:33 AMuser
11/04/2021, 8:13 AMuser
11/04/2021, 8:16 AMgunu
11/04/2021, 8:29 AMI ask because (…) isn’t showing up anymore. and i dont see the dbt logs.The first time Normalization 0.30.24+ runs, it will detect the missing
_airbyte_ab_id
column on normalized tables and should automatically decide to perform a dbt full refresh (same behavior as older normalization versions < 0.30.24) at least once
The next time it runs, the column should now exists and populated, so it can keep on running incrementally with insert or merge statements, so you should not see that “warning” anymore
is this what you are seeing?
so i’m guessing it is working without needing to do a full refresh?(full_refresh term can be confusing with an “airbyte full_refresh” where the whole table is transferred through network again, but this is not required)
user
11/04/2021, 8:41 AM_airbyte_ab_id
now? (it should start getting different values for _airbyte_normalized_at
too)
2. if you browse the workspace folder and look at the final sql files, you should see merge (or insert) statements instead of create table: https://docs.airbyte.io/operator-guides/transformation-and-normalization/transformations-with-sql#export-plain-sql-files
3. reading dbt logsuser
11/04/2021, 11:52 AM_airbyte_ab_id
is in the tables (without resetting the connector, obviously dbt realised to perform full-refresh)
also my point on dbt logs being removed. what i mean is they longer surface in the UI logs. previously we would see something like
22:21:59 | 1 of 11 OK created incremental model my_schema.my_table [SELECT in 91.02s]
but that’s not the end of the world. although would prefer to see what tables dbt creates in the UI logsuser
11/04/2021, 12:10 PMuser
11/04/2021, 12:24 PMalso my point on dbt logs being removed. what i mean is they longer surface in the UI logs. previously we would see something like
222159 | 1 of 11 OK created incremental model my_schema.my_table [SELECT in 91.02s]
but that’s not the end of the world. although would prefer to see what tables dbt creates in the UI logsmmm this user is also sharing airbyte sync logs where I was expecting to see the dbt logs too… That’s troublesome if they are not shown anymore This might need further investigation… https://airbytehq.slack.com/archives/C01MFR03D5W/p1636023601097400?thread_ts=1636017154.089100&cid=C01MFR03D5W
user
11/04/2021, 12:25 PMalso does this feature not work for mysql --> snowflake? (reading through the PR)It should, please let me know if you get into errors there
user
11/04/2021, 12:41 PMuser
11/04/2021, 12:59 PM