We have a Flink job that subscribes to a topic (wi...
# troubleshooting
v
We have a Flink job that subscribes to a topic (with 10+k/sec volume) and applies tumbling windows with aggregation to create hourly buckets; each window contains a simple counter and also a HyperLogLog structure to track uniqueness estimates. We’re using the RocksDB backend. The application should be fairly simple, but it is very slow with the RocksDB backend because the windowing operation has to read, deserialize, update, serialize and write back to disk for every incoming element, which is incredibly wasteful. With HashMap backend, it works orders of magnitude faster. Is there a way to optimize how it runs on RocksDB and reduce state reads/writes in the window? For example, process incoming events in batches rather than one at a time?
a
RocksDB only writes to disk when needed, e.g. if it has used all available memory. So check that it has sufficient memory allocated. It doesn't use JVM heap, so you need to allocate managed memory dedicated to RocksDB. See https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/#memory-management If that still doesn't help you can consider using appending only state in your operations (states the implements the
org.apache.flink.api.common.state.AppendingState
interface e.g.
ListState
). This saves you the read and deserialization steps