HI Team,
I have one use case where I have to call ...
# troubleshootingn
Nishant Goenka
08/24/2023, 2:58 PMHI Team,
I have one use case where I have to call some api to retrieve some data to enrich my input event data stream, in most of the cases data is common. So I wanted to have some caching mechanism to hold the repeated data. I have used flink states to do the same. but as I was using asyncIO for data enrichment using rest api. I have to call write total three function like below:
keyedprocessfunction---->asyncIO---->KeyedCoProcessFunction
I have multiple api calls(around 15), so for API call I had to write 3 functions. now the code has gone crazy, really not readable and non maintable. Any idea of better caching strategy? Should I move to redis or something instead of using flink states.
Any help would be really appreciated. Thanks in advance.
Nishant Goenka
08/25/2023, 4:24 AMAny help would be really appreciated.
b
Brad Murry
11/20/2023, 5:17 PMOld question, but thought I’d share a pattern I prefer to inlined enrichments:
Brad Murry
11/20/2023, 5:22 PMHere, an enrichment request is sent via side-output, which is picked up by some enrichment service and eventually pushes that enriched data back into the original stream to be joined by the keyed process function.
Makes it nice when multiple enrichments from various services may be required, easy to scale there.
There’s another variation that sends the incomplete event to the enrichment channel so that state of incomplete events becomes unnecessary.
Brad Murry
11/20/2023, 5:25 PMIn some processes, we get value from incomplete events, so we sink them as they come through, and update as enrichments are collected.
Depending on your data flow patterns, you may want to cache enrichments in state as they come in and/or window them as they go out so bursts of events that require the same enrichment(s) don’t translate to bursts of enrichment requests.
Brad Murry
11/20/2023, 5:32 PMAlso sharing for feedback from others- we’ve had good perf with this approach but always keen to hear how others solve similar issues
n
Nishant Goenka
11/22/2023, 4:29 AMThanks @Brad Murry
👍 1