Is this your first time deploying Airbyte No OS Version Inst

Question

Is this your first time deploying Airbyte No OS Version Instance Mac OS 11 5 2 Memory Disk 32 GB Deployment Docker Airbyte Version 0 35 10 alpha Source name version Twilio 0 1 2 Destination name version S3 0 2 7 Description Tables `messages` `calls` `message media` using the Incremental | Append sync mode cannot have its date columns `date sent` `date created` `date updated` queried When we try to query `date created` directly we receive a `Spectrum Scan Error` When we try `date created member0` we receive `relations date created does not exist` Moving to a Full Refresh | Overwrite sync mode which does work as evidenced by some of our other tables isn t possible because we have 4+ million records in the `messages` table

Akshay Saini · Answer

< Marcos Marx Airbyte > looks like this may have been overlooked Any insights

Justin Reynolds · Answer

Hey can you help with these info 1 date created gt type of the field 2 date created gt can you share some example value for this

Ofek Katriel · Answer

1 it appears that it comes in as different field types depending on the day but we can t say for sure because we just get a Spectrum Scan error when we crawl from our S3 bucket Sometimes it shows up as a simple string other times it shows up as a map of two strings 2 Attached is a sample output when we set a query limit of 10 for a non airbyte Twilio integration we currently use

Ofek Katriel · Answer

Hi < Nicolas Smith> could you try downloading one of the parquet file and read it with a different tool than spectrum The date fields should be stored as strings but it looks like spectrum tries to read them as a struct

Ofek Katriel · Answer

< Kevin Wood>

Vinod Varma · Answer

< Augustin Lafanechere Airbyte > From a parquet file loaded 2 14 then another loaded from November

Vinod Varma · Answer

All this data was loaded in the same run I believe yet the `date created` and other `date ` fields weren t loaded in a consistent format