Hello everyone, we are looking to build Feature Store for our org. We are looking to use Flink streaming job as the processing engine to compute the features. I am new to flink so I am probably missing something basic. Lets say I want to store count distinct ips per user in last 90 days. So this is a sliding window function and Flink uses state store to keep internal info to be able to keep updating the new ip count as and when events come in. Now lets say I want to add a new feature that says Unique devices per user in last 180 days, now If I add this new field , it would start computing from now on. How do I backfill the state store with this additional info so that it computes data correctly. Source for flink is Kafka ( that has 1 day worth of data) We have alternate source that has 5 years worth of data on S3 which can be used for back fill. But I do not know how to modify the flnk state store to be able to use this information

I’m not an expert either, but my 2 cents. I think that being able to backfill historic data is not a very central streaming use case. The purist answer is probably that you can add the state and it’ll start accumulating from the time it’s on and eventually be accurate in 180 days

If you need to bootstrap your state, I would probably look into the Hybrid Source https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/hybridsource/

(BTW I’m not sure if 180 days is something you wanna keep in window state, especially if the state can get large with time. If you’re not constantly ingesting historic data it might be a decent solution to store a smaller interval in which you’re likely to see new data and keep older data in some sink, then have queries over time periods)

Thanks @Martijn Visser I did look at that, It seems very useful when I have an empty state. But here I have a state ( that has 1 column being aggregated) and now I am looking to add another one.

I would probably not expose state directly from Flink, but sync it to a system where the features are actually stored

@Daniel Hristov I think so too , this is probably not a streaming use case, but flink is the closest to what we are looking for as feature store engine. Like the feature store itself can be redis ( sagemaker feature store etc ) The engine is something that can add new features every week, feature being another metric, count or count distinct etc. So this is going to be continuous where we keep adding new metrics as we build new requirements. Flink SQL interface gives a clean semantics to be able to construct this. Flink also gives the ability to maintain the required events in memory to be able to compute the given metrics. We would probably go to 180 days+ but I read that flink optimizes the state store by keeping track of only relevant events required for the computation of metric so belive we should be good there.

A couple of thoughts about this: When the use case is computing features for ML models, accurate estimates can be just as good as exact results. I mention this because using hyperloglog to estimate count-distinct is much cheaper than doing a true count-distinct, and it scales much better too. If the feature store is external to Flink, then you can run the live data through a Flink job running in streaming mode, and use the same job running in batch mode to do the backfill of historic data.

+1 on Hyper log log, but the use case is more than just count distinct. The idea was to be able to leverage Flink computation able to deal with a lot of internal things. Same here, instead of me implementing externally something on Feature store. I would eventually to use Flink to do approx count distinct. I am not sure if I keep the state outside how can flink help me do the heavy lifting. For ex with flink I would then store the ips and have another system to compute distincts( or approx ones ) on top of that. I am looking to be able to use Flink's system to handle that