hey we are trying to do csv (HTTPS) -> bigquery using

file

connection. There is a

number

field in the csv data and when we are pushing to

bigquery

this is taken as null. can someone help me with this ?

Is this field showing as a

number

field in the configuration for your CSV file source?

Is it possible to share the CSV line that is causing this problem?

if we discover schema it shows up as number, so I updated the schema manually using API even then it is showing as null

What is the type of the BQ column?

initally it was float then I changed it to string

ah that’s probably it then

it’s trying to write the value using the incorrect type

did you change it to string in BQ before syncing or after syncing?

I changed it before syncing

are you syncing into an existing table then? not one created by airbyte?

generally we recommend having airbyte create the tables and write to them. then after that point users often use DBT to combine/modify schemas

I have created the whole thing using API 1. Created source 2. Created connection (in this instead of using discover schema I manually added schema as string) 3. After this I clicked sync now

We don’t actually support that now for basic normalization

One sec, checking something.

So what’s happening here is the source is producing some record with data

{'key':100}

because it thinks it’s producing a number. Then at the destination and normalization level it’s getting a catalog with

string

and within normalization it isn’t actually receiving the correct value and doesn’t cast automatically.

what is the solution?

Type coercion from files is a very valid use case. I think what we might want to do is have

source-file

look at the catalog it is provided and cast the records it produces for a stream to the type specified.

Actually after thinking about it more it probably makes more sense for normalization to handle the casting in a way that doesn’t end up with a null here.

I’m going to try to reproduce this issue locally and create a ticket for fixing this.

In the meantime is it possible to use floats on the BQ side?

First I tried float in BQ but it didn't work