This message was deleted.

Hey! So, we don’t have an example with Feast specifically, but we’d love it if you contributed one. Its a pretty natural integration point 🙂 This tutorial should get you started — https://github.com/DAGWorks-Inc/hamilton/blob/0254a4e279d8afe29e517ff83497d4f62311c866/examples/feature_engineering_multiple_contexts/README.md. Basically it uses Hamilton in a few ways: 1. Transform any data offline and store it in the feature store 2. Transform any data you need online after getting it out from a feature store 3. Overrides of feature store value to handle statistics (see the

mean_age

node/how its overridden). So, Hamilton isn’t a feature store but it can be used to feed data into one/extract data out of one. You just have to be a little careful about this so you get the right data in. The offline/online connection is “backfilling”, and its a naturally tricky problem. Hamilton won’t solve it for you, although there are common techniques of updating a “high watermark” — e.g. a time stamp that says when the latest offline/online data is. Then when you run your offline job every day, you need to update your online job as well. If your streaming job just stops, you’ll probably want to run it so it keeps up (kafka has tooling to do this naturally). Another common approach is to consolidate offline/online at some time (use offline at the end to sweep over then merge with your online stuff to generate it).

@Игорь Хохолко yeah welcome to the challenges of offline & online featurization! To add to what @Elijah Ben Izzy said, what you should do here depends on SLAs and tolerance for features being stale on the online side. Something to consider that can help: • a “kappa architecture” where you can feed both offline and online from the same event stream. Happy to chat more as you think through it. I think Hamilton in front of Feast is a pretty reasonable pattern. You’d just want to make sure your read/write pattern works for your online use case to make sure Feast is a fit there.