Hey folks, got two questions:
1. We currently use ...
# ask-community-for-troubleshootingp
Phil Marius
08/31/2021, 2:14 PMHey folks, got two questions:
1. We currently use a MySQL DB as our warehouse with dbt transforming the data. I set up airbyte -> redshift last week and most tables are present, aside from the dbt transformed tables. Has anyone come across this before?
2. Airbyte is eating up a lot of resources on our Redshift instance, we’ve used ~$300 in a week with only the MySQL connection running. The same happened when we ran a trial with Snowflake, eating up ~50 credits of our 100 credit trial. Is there a way we can reduce the warehouse usage at all? Fivetran had a footprint MUCH smaller than Airbyte’s on our Snowflake trial
👀 1
j
Jeff Crooks
08/31/2021, 2:23 PMHey @Phil Marius - What is causing the cost increase in Redshift and what instance type are you on? You pay hourly for Redshift, so I wouldnt expect any service to increase costs specifically
p
Phil Marius
08/31/2021, 2:43 PMFrom the looks of things, it’s all dbt transformations running on the cluster itself. Been some pretty long running queries and Airbyte is the only thing connected to Redshift right now. Added Metabase this morning but that’s it
Phil Marius
08/31/2021, 2:48 PMOh and instance type is ra3.xplus with 2 nodes
g
George Claireaux (Airbyte)
08/31/2021, 3:01 PM
Couple of related issues around improving Airbyte's normalization with DBT that might be worth following:
https://github.com/airbytehq/airbyte/issues/2566
https://github.com/airbytehq/airbyte/issues/4286
George Claireaux (Airbyte)
08/31/2021, 3:03 PM
As for limiting resource usage right now, you could turn off Airbyte's normalization / use a custom dbt normalization to optimise for your data.
👍 2
George Claireaux (Airbyte)
08/31/2021, 3:04 PM
Hmm, although your missing tables in point
1p
Phil Marius
08/31/2021, 3:51 PMYeah course, where do I get them again?
g
George Claireaux (Airbyte)
08/31/2021, 3:52 PM
George Claireaux (Airbyte)
08/31/2021, 3:52 PM
Are you using UI?
p
Phil Marius
08/31/2021, 4:03 PMYeah UI, having a look now, seems to be a number of errors in there, can’t share but lemme find the errors
Phil Marius
08/31/2021, 4:04 PM Copy code
Invalid value: null used for required field: "updatedAt", schema type: STRINGPhil Marius
08/31/2021, 4:04 PMA lot of that
Phil Marius
08/31/2021, 4:04 PMAh, I think that’s a date column, had an issue with MySQL and dates before here https://airbytehq.slack.com/archives/C021JANJ6TY/p1629452774231800?thread_ts=1629449191.229900&cid=C021JANJ6TY.
Could it be related?
g
George Claireaux (Airbyte)
08/31/2021, 4:40 PM
Very possible! If you try adding that param Chris suggests in that thread you might have some luck
p
Phil Marius
09/01/2021, 3:15 PMAlready have that, still failing
Phil Marius
09/01/2021, 3:15 PMWill share logs
Phil Marius
09/01/2021, 3:25 PMWill DM you