https://linen.dev logo
g

gunu

11/04/2021, 1:56 AM
1. do you need to perform a full-refresh in order to take advantage of the new incremental normalization? (apologies in advance if this is in the docs somewhere). i assume so given
_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh
but just want to confirm there’s no workaround as its a pretty big task 2. are the dbt logs removed?
u

user

11/04/2021, 3:12 AM
@Chris (deprecated profile) can you help here?
u

user

11/04/2021, 6:30 AM
i ask because
_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh
isn’t showing up anymore. and i dont see the dbt logs. so i’m guessing it is working without needing to do a full refresh?
g

gunu

11/04/2021, 6:33 AM
also this is MySQL source to Snowflake destination
u

user

11/04/2021, 8:13 AM
To populate the airbyte ab id column, yes, it needs a dbt full refresh (not an airbyte full refresh) So just recreating the normalizing tables from the airbyte raw tables instead of inserting incrementally
u

user

11/04/2021, 8:16 AM
Dbt logs are not removed, they should be in the workspace where the sync runs from, see https://docs.airbyte.io/operator-guides/browsing-output-logs#exploring-the-logs-folders
g

gunu

11/04/2021, 8:29 AM
I ask because (…) isn’t showing up anymore. and i dont see the dbt logs.
The first time Normalization 0.30.24+ runs, it will detect the missing
_airbyte_ab_id
column on normalized tables and should automatically decide to perform a dbt full refresh (same behavior as older normalization versions < 0.30.24) at least once The next time it runs, the column should now exists and populated, so it can keep on running incrementally with insert or merge statements, so you should not see that “warning” anymore is this what you are seeing?
so i’m guessing it is working without needing to do a full refresh?
(full_refresh term can be confusing with an “airbyte full_refresh” where the whole table is transferred through network again, but this is not required)
u

user

11/04/2021, 8:41 AM
You can verify the normalization/dbt incremental behavior by: 1. does your final normalized tables include the
_airbyte_ab_id
now? (it should start getting different values for
_airbyte_normalized_at
too) 2. if you browse the workspace folder and look at the final sql files, you should see merge (or insert) statements instead of create table: https://docs.airbyte.io/operator-guides/transformation-and-normalization/transformations-with-sql#export-plain-sql-files 3. reading dbt logs
u

user

11/04/2021, 11:52 AM
ahhh dbt full-refresh! i was thinking full-refresh = resetting the connector. that makes sense (i’m familiar with dbt incremental feature) confirming
_airbyte_ab_id
is in the tables (without resetting the connector, obviously dbt realised to perform full-refresh) also my point on dbt logs being removed. what i mean is they longer surface in the UI logs. previously we would see something like
Copy code
22:21:59 | 1 of 11 OK created incremental model my_schema.my_table [SELECT in 91.02s]
but that’s not the end of the world. although would prefer to see what tables dbt creates in the UI logs
u

user

11/04/2021, 12:10 PM
also does this feature not work for mysql --> snowflake? (reading through the PR)
u

user

11/04/2021, 12:24 PM
also my point on dbt logs being removed. what i mean is they longer surface in the UI logs. previously we would see something like
222159 | 1 of 11 OK created incremental model my_schema.my_table [SELECT in 91.02s]
but that’s not the end of the world. although would prefer to see what tables dbt creates in the UI logs
mmm this user is also sharing airbyte sync logs where I was expecting to see the dbt logs too… That’s troublesome if they are not shown anymore This might need further investigation… https://airbytehq.slack.com/archives/C01MFR03D5W/p1636023601097400?thread_ts=1636017154.089100&amp;cid=C01MFR03D5W
u

user

11/04/2021, 12:25 PM
also does this feature not work for mysql --> snowflake? (reading through the PR)
It should, please let me know if you get into errors there
u

user

11/04/2021, 12:41 PM
will do. also on the logs from. a lot of different connectors the logs aren’t coming through e.g. i have a GSC connector and the logs aren’t being surfaced in the UI but when i follow the logs in cloudwatch its all there
u

user

11/04/2021, 12:59 PM
Could you create a github ticket to address disappearing logs in the UI please?
2 Views