Rajat
04/30/2025, 11:09 AMXiang Fu
Rajat
04/30/2025, 11:27 AMRajat
04/30/2025, 11:28 AMXiang Fu
created_at as the timestamp, then your upsert realtime table and offline table won’t have duplicates right>Xiang Fu
Rajat
04/30/2025, 12:26 PMRajat
04/30/2025, 12:29 PMRajat
04/30/2025, 12:30 PMRajat
04/30/2025, 12:39 PMXiang Fu
s_created_at_day < today-3d and realtime table is applying s_created_at_day >= today-3d , what would be the issue?Xiang Fu
Rajat
04/30/2025, 1:23 PMRajat
04/30/2025, 1:24 PMs_created_at_day < today-3d and realtime table is applying s_created_at_day >= today-3d ,
"
how does Pinot manages that?Rajat
04/30/2025, 1:33 PMXiang Fu
Xiang Fu
Xiang Fu
Rajat
04/30/2025, 1:44 PMXiang Fu
Rajat
04/30/2025, 1:45 PMs_created_at older than 3 daysXiang Fu
s_created_atRajat
04/30/2025, 1:45 PMs_created_atXiang Fu
s_created_at > now() - 3dXiang Fu
SELECT *
FROM tableName
WHERE s_created_at > ago('P3D')Rajat
04/30/2025, 1:47 PMXiang Fu
Xiang Fu
Xiang Fu
Rajat
04/30/2025, 1:49 PMRajat
04/30/2025, 1:50 PMXiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
Rajat
04/30/2025, 1:53 PMXiang Fu
Rajat
04/30/2025, 1:53 PMXiang Fu
Xiang Fu
Rajat
04/30/2025, 1:55 PMRajat
04/30/2025, 1:56 PMXiang Fu
Rajat
04/30/2025, 1:56 PMXiang Fu
WHERE s_created_at > ago('P3D')Xiang Fu
Xiang Fu
Rajat
04/30/2025, 1:59 PMWHERE s_created_at > ago('P3D') I want this filter just so that my data doesn't club between offline and online while adding dataXiang Fu
select count(*) from myTable where s_created_at > ago('P3D') and comparing it with ``select count(*) from myTable` so the time difference is the extra latency for your predicateXiang Fu
Xiang Fu
Xiang Fu
s_created_at in both table configs for timeColumnNameRajat
04/30/2025, 2:01 PMXiang Fu
Xiang Fu
Rajat
04/30/2025, 2:02 PMXiang Fu
Xiang Fu
s_created_at comes from?Rajat
04/30/2025, 2:04 PMRajat
04/30/2025, 2:04 PMs_created_at_dayRajat
04/30/2025, 2:04 PMRajat
04/30/2025, 2:05 PMXiang Fu
Rajat
04/30/2025, 2:06 PMRajat
04/30/2025, 2:07 PMXiang Fu
Rajat
04/30/2025, 2:09 PMXiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
SELECT * FROM myTable_OFFLINE WHERE s_created_at_day < $TIME_BOUNDARY
UNION
SELECT * FROM myTable_REALTIME WHERE s_created_at_day >= $TIME_BOUNDARYXiang Fu
Xiang Fu
select count(*) from myTable , the underlying logic to compute is similar to this:
select count(*) from
(
SELECT * FROM myTable_OFFLINE WHERE s_created_at_day < $TIME_BOUNDARY
UNION
SELECT * FROM myTable_REALTIME WHERE s_created_at_day >= $TIME_BOUNDARY
)Rajat
04/30/2025, 2:16 PMRajat
04/30/2025, 2:16 PMRajat
04/30/2025, 2:17 PMRajat
04/30/2025, 2:33 PMs_created_at the timeColumn that is coming from kafka. It will work the same right?Rajat
04/30/2025, 2:34 PMs_created_at_dayXiang Fu
Xiang Fu
s_created_at_day is only in REALTIME table not offline table?Rajat
04/30/2025, 3:02 PMRajat
04/30/2025, 3:02 PMXiang Fu
Xiang Fu
Rajat
04/30/2025, 3:06 PMRajat
04/30/2025, 4:01 PMXiang Fu
Xiang Fu
Rajat
04/30/2025, 4:35 PM