Hello! I am a new in Flink and I need help of community.
Introductory:
- We have a topic with 12 partitions
- Producer writes 50k events per second to it
Partition key - userID, we guarantee that user events are processed in the order they were created
Events - JSONs of 2 types,
1. JSONs which need to be enriched with a state of a user which send it (every event had user_id field)
2. JSON with rules how to change the state of a user
Enriched events are placed in another topic and go further along the pipeline
State of one user - dictionary, stored as N rows in Cassandra.
Unique users - 30 000 000
Average count of properties in user - 300
We have microservices are written in Golang for all steps of pipeline
Running in k8s
------------
Questions:
• How configure flink for enriching events with state from cassandra? Should I use statefun for this case?
• How many messages per second is one flink instance can handle? How to calculate desired amount of hardware?
• Do I need to change the number of partitions?
• How to achieve low latency with Apache Flink? Should I rewrite microservice to Java/Scala or not?
• Things I should think/avoid before integrate Flink in Production?
Any help or ideas appreciated 🙂