Is it possible to use the Flink CEP library on its...
# randomt
Tyron
02/20/2023, 12:02 PMIs it possible to use the Flink CEP library on its own? The Flink CEP patterns are implemented as a state machine, can a sequence of objects be tested against this pattern outside the Flink context?
n
Nicholas Jiang
02/20/2023, 12:54 PMYou could refer to
org.apache.flink.cep.nfa.compiler.NFACompilerTestt
Tyron
02/20/2023, 12:56 PMI was actually looking at that, and got the idea to use it from that class. But not sure if this is intended to be used in this way. I'm not seeing an @Internal annotation or anything, so guess it can be used,
Tyron
02/20/2023, 12:57 PMWill give it a shot, thanks @Nicholas Jiang
n
Nicholas Jiang
02/20/2023, 1:00 PMflink-cep module has little annotation. You could regard the interface as public.
Nicholas Jiang
02/20/2023, 1:01 PMBTW, if you have any question on CEP, I will give you some answer and suggestion.
🙏 1
t
Tyron
02/20/2023, 1:04 PMMy main question is how can I add and remove CEP patterns dynamically? @Nicholas Jiang
As far as I know, this is not yet supported in Flink.
Hence, my approach so far has been to use Flink to read 2 sources : my business events, and new patterns. Then I'm using a broadcast to broadcast patterns to all operator instances, and then have an operator that is calling an inhouse built pattern matcher on the business events. I do not like building my own pattern matcher when Flink CEP already exists, so seeing how I can use this library in a dynamic way.
Tyron
02/20/2023, 1:04 PMBy dynamic I mean add and remove patterns without redeploying.
n
Nicholas Jiang
02/20/2023, 1:09 PMYou could follow the FLIP-200, in which proposal the pattern is serializable JSON string.
Nicholas Jiang
02/20/2023, 1:11 PMIMO, you could modify the
PatternSerializableSerializableNicholas Jiang
02/20/2023, 1:14 PMBTW, you could refer to https://github.com/RealtimeCompute/ververica-cep-demo.
t
Tyron
02/20/2023, 1:30 PMThanks for this information.
Tyron
02/20/2023, 1:30 PMSo from my understanding, most of the work will be to replicate what CEPOperator is doing to handle more than 1 pattern, by keeping a list of NFA and NFA states and iterate over them for each event.
n
Nicholas Jiang
02/20/2023, 1:32 PMRight. Like
PatternProcessorOperatorPatternPatternProcessorCoorindatort
Tyron
02/20/2023, 1:51 PMLooks like the ververica is a commercial feature and doesn't have the entire codebase.
Tyron
02/20/2023, 1:51 PMWill give this a shot, and see how it goes. Thanks for your help.
n
Nicholas Jiang
02/20/2023, 1:53 PMYes. But the brief changes are mentioned in FLiP-200. You can implement according to this by yourself, which works are small.
y
YAO WANG
02/22/2023, 7:56 AM@Nicholas Jiang thanks, very helpful!
t
Tyron
02/25/2023, 4:42 PMIs there any implementation documentation available somewhere for CEPOperator? Seeing all kinds of states being maintained and updated - not sure I understand the code well. @Nicholas Jiang
n
Nicholas Jiang
02/26/2023, 5:48 AMNo. What is your understand for CEPOperator？
t
Tyron
02/26/2023, 7:56 AMI do not really know what is the difference between the computationStates and partialMatches? Do I have to keep both in a my custom oerpator?
n
Nicholas Jiang
03/02/2023, 5:44 AMComputationStates stores the matched result and partialMatches stores the partial result of pattern match.
5 Views