https://linen.dev logo
p

Preet Singh

02/16/2022, 7:27 PM
Is this your first time deploying Airbyte: No OS Version / Instance: Mac OS 11.5.2 Memory / Disk: 32 GB Deployment: Docker Airbyte Version: 0.35.10-alpha Source name/version: Twilio 0.1.2 Destination name/version: S3 0.2.7 Description: Tables (
messages
,
calls
,
message_media
) using the "Incremental | Append" sync mode cannot have its date columns (
date_sent
,
date_created
,
date_updated
) queried. When we try to query
date_created
directly, we receive a
Spectrum Scan Error
. When we try
date_created.member0
we receive
relations "date_created" does not exist
. Moving to a "Full Refresh | Overwrite" sync mode (which does work as evidenced by some of our other tables) isn't possible because we have 4+ million records in the
messages
table
a

Akshay Saini

02/17/2022, 5:36 PM
@Marcos Marx (Airbyte) looks like this may have been overlooked. Any insights?
j

Justin Reynolds

02/18/2022, 4:36 AM
Hey can you help with these info 1. date_created -> type of the field? 2. date_created -> can you share some example value for this
o

Ofek Katriel

02/21/2022, 5:45 PM
1. it appears that it comes in as different field types depending on the day, but we can't say for sure because we just get a Spectrum Scan error when we crawl from our S3 bucket. Sometimes it shows up as a simple string, other times it shows up as a map of two strings 2. Attached is a sample output when we set a query limit of 10 for a non-airbyte Twilio integration we currently use
Hi @Nicolas Smith, could you try downloading one of the parquet file and read it with a different tool than spectrum? The date fields should be stored as strings but it looks like spectrum tries to read them as a struct
@Kevin Wood
v

Vinod Varma

02/21/2022, 7:01 PM
@Augustin Lafanechere (Airbyte) From a parquet file loaded 2/14, then another loaded from November:
All this data was loaded in the same run I believe, yet the
date_created
and other
date_*
fields weren’t loaded in a consistent format.