Hey folks, got two questions: 1. We currently use ...
# ask-community-for-troubleshooting
Hey folks, got two questions: 1. We currently use a MySQL DB as our warehouse with dbt transforming the data. I set up airbyte -> redshift last week and most tables are present, aside from the dbt transformed tables. Has anyone come across this before? 2. Airbyte is eating up a lot of resources on our Redshift instance, we’ve used ~$300 in a week with only the MySQL connection running. The same happened when we ran a trial with Snowflake, eating up ~50 credits of our 100 credit trial. Is there a way we can reduce the warehouse usage at all? Fivetran had a footprint MUCH smaller than Airbyte’s on our Snowflake trial
👀 1
Hey @Phil Marius - What is causing the cost increase in Redshift and what instance type are you on? You pay hourly for Redshift, so I wouldnt expect any service to increase costs specifically
From the looks of things, it’s all dbt transformations running on the cluster itself. Been some pretty long running queries and Airbyte is the only thing connected to Redshift right now. Added Metabase this morning but that’s it
Oh and instance type is ra3.xplus with 2 nodes
Couple of related issues around improving Airbyte's normalization with DBT that might be worth following: https://github.com/airbytehq/airbyte/issues/2566 https://github.com/airbytehq/airbyte/issues/4286
As for limiting resource usage right now, you could turn off Airbyte's normalization / use a custom dbt normalization to optimise for your data.
👍 2
Hmm, although your missing tables in point
is odd. Could you share sync logs?
Yeah course, where do I get them again?
Are you using UI?
Yeah UI, having a look now, seems to be a number of errors in there, can’t share but lemme find the errors
Copy code
Invalid value: null used for required field: "updatedAt", schema type: STRING
A lot of that
Ah, I think that’s a date column, had an issue with MySQL and dates before here https://airbytehq.slack.com/archives/C021JANJ6TY/p1629452774231800?thread_ts=1629449191.229900&cid=C021JANJ6TY. Could it be related?
Very possible! If you try adding that param Chris suggests in that thread you might have some luck
Already have that, still failing
Will share logs
Will DM you