Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

hello! New to airbyte, great tool btw. I do have a question. I have setup a connection from a online csv to a local postgres. I am unsure how to change the data type and field names of the source before it hits the destination?

Airbyte will mirror the source name for columns and try to infer the data type, at the moment you cannot change the type selected from Airbyte. You can, as already mentioned, use custom dbt to post-processed your data

this is an *advanced* usage of source-file:

in the docs of the source-file connector, the last example shows how to rename field names from the CSV file though:
<https://docs.airbyte.io/integrations/sources/file>

it’s similar to this blog post (and specific to this connector since in Airbyte in general, you can’t rename or change types yet as <@U01MMSDJGC9> mentionned)
<https://www.kite.com/python/answers/how-to-set-column-names-when-importing-a-csv-into-a-pandas-dataframe-in-python>

it should also be possible to change types:
<https://www.kite.com/python/answers/how-to-specify-dtype-when-reading-a-csv-in-python>

If you make use of the custom dbt transformation to operate on the destination data, you can change column names and types of course though!

If I use the default parameters with the `epidemiology` dataset example in my source, I get these types (noticed the number columns):

But if i change the reader_options to `{"dtype":"string"}` , i can force all columns to be parsed as strings…
which results in the following schema:

&gt; *dtype:* Type name or dict of column -&gt; type, optional
&gt; Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32, ‘c’: ‘Int64’} Use str or object together with suitable na_values settings to preserve and not interpret dtype. If converters are specified, they will be applied INSTEAD of dtype conversion.
from <https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html>

Is it possible to specified only for one colums the dtype ?

Yes, it’s in the pandas docs as i quoted:
&gt; E.g. {‘a’: np.float64, ‘b’: np.int32, ‘c’: ‘Int64’}

Ok thanks !! It worked ! I've been looking into this problem for 2 days haha