Hi team :slightly_smiling_face: I'm using Airbyte ...
# replication-troubleshooting
a
Hi team 🙂 I'm using Airbyte open-source, and I'm trying to import a csv file from s3 to Redshift. I currently have a problem that I can't figure out: one of my fields is a date in the CSV and I would like to ingest it as it in my Redshift table. But I can't find the right way to do it 😕, any help would be appreciated! And example of the input file:
Copy code
ID,USER_ID,ORDER_DATE,STATUS
1,1,2018-01-01,returned
2,3,2018-01-02,completed
3,94,2018-01-04,completed
4,50,2018-01-05,completed
5,64,2018-01-05,completed
n
Hi, Alexandre! Do I understand correctly that the date type changes after ingestion into Redshift? Could you give me an example of an output file? Sometimes type conversions happen between connectors, in that case I'd say the best thing to do is use a dbt transformation. We've got some tutorials in the docs!
a
Yes, but because I can't find a way to say that
ORDER_DATE
is a date in the Airbyte CSV connector:
Copy code
ID,USER_ID,ORDER_DATE,STATUS
1,1,2018-01-01,returned
2,3,2018-01-02,completed
3,94,2018-01-04,completed
4,50,2018-01-05,completed
5,64,2018-01-05,completed
And currently the table created on Redshift is :
Copy code
CREATE TABLE tmp_data.jaffle_shop_orders (
    id character varying(65535) ENCODE lzo,
    status character varying(65535) ENCODE lzo,
    user_id character varying(65535) ENCODE lzo,
    order_date character varying(65535) ENCODE lzo,
    _ab_source_file_url character varying(65535) ENCODE lzo,
    _ab_additional_properties super,
    _ab_source_file_last_modified timestamp with time zone ENCODE az64,
    _airbyte_ab_id character varying(256) ENCODE lzo distkey,
    _airbyte_emitted_at timestamp with time zone ENCODE raw,
    _airbyte_normalized_at timestamp without time zone ENCODE az64,
    _airbyte_jaffle_shop_orders_hashid character varying(32) ENCODE lzo
)
DISTSTYLE AUTO
SORTKEY ( _airbyte_emitted_at );
I would like to have the order date on the redshift side with the type
TIMESTAMP
if possible.
n
Oh gotcha! Thanks for all the details. Have you been able to look into dbt? Here's just one of the docs we have on it: https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt/