Hello! I am a new in Flink and I need help of comm...
# troubleshooting
e
Hello! I am a new in Flink and I need help of community. Introductory: - We have a topic with 12 partitions - Producer writes 50k events per second to it Partition key - userID, we guarantee that user events are processed in the order they were created Events - JSONs of 2 types, 1. JSONs which need to be enriched with a state of a user which send it (every event had user_id field) 2. JSON with rules how to change the state of a user Enriched events are placed in another topic and go further along the pipeline State of one user - dictionary, stored as N rows in Cassandra. Unique users - 30 000 000 Average count of properties in user - 300 We have microservices are written in Golang for all steps of pipeline Running in k8s ------------ Questions: • How configure flink for enriching events with state from cassandra? Should I use statefun for this case? • How many messages per second is one flink instance can handle? How to calculate desired amount of hardware? • Do I need to change the number of partitions? • How to achieve low latency with Apache Flink? Should I rewrite microservice to Java/Scala or not? • Things I should think/avoid before integrate Flink in Production? Any help or ideas appreciated 🙂