i could modify the split, but evidently you’re not supposed to do that - you’re supposed to modify the SplitState. so i’m not sure what this could actually mean, given that the SplitReader gets Splits, and not SplitStates…

hey Trystan, did you ever find a path forward? we’re having stability issues with a long running batch job and we want to store the state in

SplitState

so that it will reenter roughly where it failed, but we have some doubts if it will actually work and you’re right, the reference cassandra source implementation kinda skirts around it

hey! it’s been awhile and that project got put on hold for a bit… but i believe i set the state/progress tracking info on the splitstate, right after emitting (so in the RecordEmitter implementation). unfortunately, i can’t remember if i tested this properly, and i didn’t get around to writing any e2e tests to validate state behavior halfway through… so i wouldn’t put too much stock into this approach. i never got a proper answer to it…

thanks for the response! maybe we’ll give it a shot. the flink docs say they don’t use checkpoints in batch mode but it vaguely says we can still use state primitives so it sounds like it might be worth a try…