Hi, To use rollup segments effectively is there an...
# troubleshooting
n
Hi, To use rollup segments effectively is there any condition we need to follow like filtering on raw timestamp column without using
Copy code
(DATE_TRUNC('DAY', "eventsgroups".servertime)) >= CAST('2025-08-25' AS timestamp)
date_trunc functions?
m
I don't fully follow the question. But if you mean lowering the granularity from say seconds to hours or DAYS, then yes. That will rollup multiple rows from the same hour/DAY into single one. This is true for rollup in any system.
n
I'm using trino driver along with dbt as semantic layer with lightdash as BI tool. when i create a chart to avg(metric) group by day. its quering all the rows instead of only looking into the rollup segments. rolluptask "1hour.mergeType": "rollup", "1hour.bucketTimePeriod": "1h", "1hour.bufferTimePeriod": "1h", "1hour.maxNumRecordsPerSegment": "1000000", "1day.mergeType": "rollup", "1day.bucketTimePeriod": "1d", "1day.bufferTimePeriod": "1d", "1day.roundBucketTimePeriod": "1d", "1day.maxNumRecordsPerSegment": "1000000", "1month.mergeType": "rollup", "1month.bucketTimePeriod": "30d", "1month.bufferTimePeriod": "1d", "1month.roundBucketTimePeriod": "30d", "1month.maxNumRecordsPerSegment": "1000000"
is my config is correct. how to effectively query only the rollup segments.
m
Have you configured minions? Is the rollup actually happening?
n
yes
m
@Manish ^^
m
Here you have specified the buffer time is 1 day, so if you query data older than 1 day, you should see rolled up segments right? Segments that have been rolled up have a prefix
merged_
in the segment name if you really want to see which segments have been rolled up.
n
i have these rollups created under the table. In trino still says it fetching all rows. from trino i found this why rows scanned is still equal to overall rows. Totally i have 9.13M rows in my table.
Copy code
| # | Query Type                                                         | CPU Time | Latency (Elapsed Time) | Rows Scanned (Input Rows) |
| - | ------------------------------------------------------------------ | -------- | ---------------------- | ------------------------- |
| 1 | `COUNT(DISTINCT machine)`                                          | 51 ms    | 2.25 s                 | 1                         |
| 2 | `COUNT(seqname)`                                                   | 681 ms   | 6.35 s                 | 1.67 M                    |
| 3 | `COUNT(DISTINCT machine), COUNT(seqname) GROUP BY DAY`             | 2.83 s   | 7.12 s                 | 1.67 M                    |
| 4 | `COUNT(seqname)` with `t2_log = 'Scheduled'`                       | 297 ms   | 4.47 s                 | 1.67 M                    |
| 5 | `COUNT(DISTINCT machine), COUNT(seqname) GROUP BY DAY` with filter | 2.92 s   | 5.48 s                 | 1.67 M                    |
| 6 | `t1_seq, COUNT(seqname) GROUP BY t1_seq (TOP N)`                   | 909 ms   | 0.71 s                 | 1.67 M                    |
| 7 | `machine, COUNT(seqname) GROUP BY machine (TOP 5)`                 | 1.14 s   | 0.84 s                 | 1.67 M                    |
| 8 | `t1_seqname, COUNT(seqname) GROUP BY t1_seqname (TOP 5)`           | 517 ms   | 0.50 s                 | 1.67 M                    |
| 9 | `COUNT(DISTINCT machine)` with `t2_log = 'Scheduled'`              | 3 ms     | 1.68 s                 | 1                         |
m
Can you run some queries on Pinot? The Trino connector mode that talks directly to server won’t have the broker logic to prefer rolled up segments