https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
k

Kris Hagel

07/15/2021, 1:52 PM
Hello, we are just getting started with Airbyte, we have some data loads going between databases and Redshift and those are great. We are running into some problems though in loading csv files from SFTP sources. The normalization process is failing on them as it is casting the fields to the wrong data types automatically. I understand that the theory behind Airbyte is really focused on the EL layers and not so much transformation, but I’m trying to figure out the easiest way to modify the field types. Is the only way within Airbyte to build a dbt transformation that the connection can use, or are their any options that are easier to just change a few fields that are automatically detected as an incorrect type? Thanks for any help you can provide.
c

Chris (deprecated profile)

07/15/2021, 1:54 PM
Is the only way within Airbyte to build a dbt transformation that the connection can use, or are their any options that are easier to just change a few fields that are automatically detected as an incorrect type?
Yes, you can do a custom transformation post-sync to choose the types yourself. I also made some more screenshots to show how to do that with the method before sync here: https://airbytehq.slack.com/archives/C021JANJ6TY/p1626360296211600?thread_ts=1625098194.040200&cid=C021JANJ6TY
k

Kris Hagel

07/15/2021, 3:12 PM
Thanks @Chris (deprecated profile) Forcing all columns to parse as strings would get us to at least get the data into fields, we would still have to deal with it again. I was pretty sure it was going to be worth our time to learn dbt transformations, I guess we better go that route because we’ll need to do that or something like it eventually.
👍 1
c

Chris (deprecated profile)

07/15/2021, 3:13 PM
What are the types that are not properly handled?
3 Views