I have a flink SQL job reading and writing from/to...
# troubleshooting
s
I have a flink SQL job reading and writing from/to kafka The schema of the input is below:
pid
string
version
string and
event_time
is the timestamp column I have a query right now to give per-minute aggregated events
Copy code
SELECT
  pid as key,
  TUMBLE_START(event_time, INTERVAL '1' MINUTE) as windowTime,
  COUNT(*) as metricsCount
FROM events
GROUP BY
  pid,
  TUMBLE(event_time, INTERVAL '1' MINUTE)
I want to add a column to this that is a map/json with
version
level counts
, so an example output of the whole query would be pid. windowTime. metricsCount. versionLevelMetricsCount 12. datetime. 24 { v1: 15, v2: 9 } I tried it but it doesn't accept the sql, mostly along the lines of "cannot send update changes ..GroupedAggregate to this sink", and then a few other things I tried didn't work as well what is the standard way to achieve this? also note that the actual logic is more complicated, but I have put a minimal example of what I want to do above In the actual logic, we have a UDF that is a "dedupe counter", so not just a simple count(*) it dedupes based on pid, and then a few other columns for that 1 minute interval, so if another event with those columns being equal come, then the counter doesn't increment
s
i would think changing to use window TVF would solve your issue, because the nature of the results emitted should change from a changelog stream to an event stream.
Unlike other aggregations on continuous tables, window aggregation do not emit intermediate results but only a final result, the total aggregation at the end of the window. Moreover, window aggregations purge all intermediate state when no longer needed.
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/