Hi! I have a streaming tableAPI program that does...
# random
m
Hi! I have a streaming tableAPI program that does some nonwindowed computations (groupbys, joins, etc) - the output is a single row value (e.g. mean of one column). Is it possible to force Flink to process data in fixed mini-batches, (e.g. of size 1000, so that for e.g. 10k dataset, I'd have 10 updates at output)? Unfortunatelly I cannot use
table.exec.mini-batch.size
as it requires setting
mini-batch.allow-latency
, and the time needed to process each mini-batch differ a lot. Also I want to have exact 1000 batch size for input only, and not for every operator in the pipeline. I tried to set up a custom watermark strategy to advance watermarks every 1000 items, but it seems these watermarks are ignored for nonwindowed operations (they seem not to perform buffering for plain groupbys). Can anyone relate to this?
It seems I got it somehow. Minibatch can run in one of two modes: ProcTime and RowTime. I was able to force Flink to use RowTime by doing something stupid: adding interval join on input table with self. Perhaps there is somewhere a less hacky way to force this...