David Bryson
08/09/2023, 11:44 PM{
"meta": { "id": 0, "ingestion_time": "time" },
"data": [{"id": 0, "sensor": 1, "timestamp": time1}, {"id": 0, "sensor": 1, "timestamp": time2}, {"id": 0, "sensor": 1, "timestamp": time3}]
}
I'm really struggling with how to create table with this input data from a stream without doing it in two queries, so my first query ends up being something like:
CREATE TABLE {0} (
meta ROW(`id` VARCHAR, `ingestion_time` INTEGER),
data ARRAY<ROW(`id` VARCHAR, `sensor` DOUBLE, `timestamp` STRING)>
)
What I actually need in my tables is one row per data array entry. So I have this second query:
SELECT items.id,
items.sensor,
TO_TIMESTAMP(replace(replace(items.timestamp, '+00:00', ''), 'T', ' '))
FROM
{1} CROSS JOIN UNNEST(data) as items (id, sensor, timestamp);
This seems really inefficient and complicated, is there a way I can improve this and hopefully put in a watermark/partitioning schema in the first query?