Hi team. I edited a table’s config setting its

realtime.segment.flush.threshold.time

to 2h. This is its streaming config:

"streamConfigs": {
        "streamType": "kafka",
        //other configs skipped for brevity
        "realtime.segment.flush.threshold.time": "2h",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
      }

I am expecting segments to commit every 2h but here’re the last few commits:

"raw_trace_view__3__918__20220217T1600Z": {
      "Server_server-realtime-5.server-realtime-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "raw_trace_view__3__919__20220217T1616Z": {
      "Server_server-realtime-5.server-realtime-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "raw_trace_view__3__920__20220217T1631Z": {
      "Server_server-realtime-5.server-realtime-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "raw_trace_view__3__921__20220217T1647Z": {
      "Server_server-realtime-5.server-realtime-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    }

Why are segments not being committed as expected? Do I need to do anything else after editing the config?

could you share which part is not as expected? it seems like it i committing, yes? did you meant they are committing faster than 2hr? (this is possible as there's also a size threshold)

@Rong R

did you meant they are committing faster than 2hr

Yes. For some reason, I was thinking that not specifying the size threshold will make it commit at the specified interval. The segments’ size is ~200M, so looks like it is picking the default value here.