Hi, I'm new to Flink and currently working on a (P...
# troubleshooting
p
Hi, I'm new to Flink and currently working on a (PoC). I have a use case where the input is JSON formatted as shown below. I want to perform a keyBy operation as indicated in the snippet and I do have other operators to convert the following json as showed below. If the key already exists, I want to discard the value. If the key does not exist, I want to sink it to Kafka. The TTL (Time To Live) for the key is one month, and the incoming logs have a high QPS around 60k. Problems: 1) I have tried using RocksDB, but the incremental checkpoint size kept increasing, eventually reaching a few gigabytes and causing checkpoints to fail. 2) If I use FS state, memory usage also keeps increasing. 3) I have also tried storing the keys in a Redis Bloom Filter, but it slowed down the entire application, resulting in increased backpressure. How should I tackle this and is there any other alternative approach ? Example input json
Copy code
{
  "timestamp": 1717047000567,
  "ssp_id": 10,
  "dsp_id": 3,
  "uuid": "123456-a09a-492b-9694-f52016c8c72c",
  "domain": "google.com",
  "cat1": 1,
  "cat2": 2,
  "cat3": 3,
  "cat4": 4
}
Code Snippet
Copy code
DataStream<Segment> uniqueSegments = segments.keyBy(value -> new Segment(
        value.uuid,
        value.doman,
        value.some_id_int,
        value.some_ID_int,
        value.cat1,
        value.cat2,
        value.cat3,
        value.cat4))
.process(new EventProcessor()).uid("some-uuid");


public class SegmentEventProcessor extends KeyedProcessFunction<Segment, Segment, Segment> {

    private static final long serialVersionUID = 1L;
    private transient ValueState<Boolean> flagState;
    private transient ValueState<Long> timerState;
    private static final long ONE_MONTH = 2592000000L;

    @Override
    public void open(OpenContext openContext) {
        ValueStateDescriptor<Boolean> flagDescriptor = new ValueStateDescriptor<>(
                "flag",
                Types.BOOLEAN);
        flagState = getRuntimeContext().getState(flagDescriptor);

        ValueStateDescriptor<Long> timerDescriptor = new ValueStateDescriptor<>(
                "timer-state",
                Types.LONG);
        timerState = getRuntimeContext().getState(timerDescriptor);
    }

    @Override
    public void processElement(Segment segment, Context context, Collector<Segment> collector) throws Exception {
        if (flagState.value() == null || !flagState.value()) {

            collector.collect(segment);
            
            flagState.update(true);
            long timer = context.timerService().currentProcessingTime() + ONE_MONTH;
            context.timerService().registerProcessingTimeTimer(timer);

            timerState.update(timer);
        }
    }

    @Override
    public void onTimer(long timestamp, OnTimerContext ctx, Collector<Segment> out) {
        timerState.clear();
        flagState.clear();
    }
}
s
Are you keying the whole object?
Copy code
keyBy(value -> new Segment(
        value.uuid,
        value.doman,
        value.some_id_int,
        value.some_ID_int,
        value.cat1,
        value.cat2,
        value.cat3,
        value.cat4))
If the key is UUID shouldnt it be?
Copy code
keyBy(value -> 
        value.uuid)
Even then, if you try to maintain all UUIDs in Flink state for a month, it will be huge.
p
1. The UUID is equivalent to the User ID. Therefore, if the same user comes from a different domain, I don't want it to be logged into the database again. 2. I am trying to optimize the Pinot database so that the data is only logged once for faster query response times. The state is indeed very large. 3. Could you share some insights on whether there is a different approach or tool that I could use? Is there a way to store only the UIDs while ignoring all the other values? @Shan P S
s
1. Check your keyBy() logic. It seems to key by each Segment object, vs UUID. 2. Also, having a timer to clear the state is not a good idea. The timer will be be adding to overhead and state. There will be one timer per key. Use TTL for state instead https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#state-time-to-live-ttl 3. Having a shorter TTL will reduce checkpoint size. 4. A DB sink after process(new EventProcessor()) with dedupe logic will improve performance. 5. You can tune 3 and 4 to suit your performance needs.
p
Hi @Shan P S I understood all the other points except point 4, which mentions that implementing deduplication logic for the database sink will improve performance. Could you please explain what you meant by this?