Hi, I'm new to Apache Flink and could use some help. I have a question: can a single TaskManager consume data from multiple Kafka topics? If so, when using RocksDB as the state backend, does it maintain state separately for each topic, or is the state shared across all topics processed by that TaskManager? Thanks in advance!

Flink is a distributed framework. The number of task-managers (TM) only dictates the level of paralyzation; each TM is configured by a number of available slots. General "rule-of-thumb" CPU == SLOTS; aka: how much concurrent threads per TM. We run TM with 4 CPU, thus configure 4 SLOTS per TM x 16 of them = 64 SLOTS (we'll horizontally scale TM up and down thus SLOTS will change). To answer your question, the number of Kafka topics reading from does not matter to the number of TMs (or even SLOTs). But what does matter is how much concurrency you'll get. Each Kafka partition will be assigned to a SLOT; if you have only one SLOT, then each partition will take turns pulling in data. However, depending on what you do with the data, pulling data in from Kafka will not the bottleneck. Scale

parallelism

appropriate to your processing speed needs. e.g: Kafka topic with 2 partitions; only max of 2 SLOTS will read data in, but once in Flink it can be distributed to N number of SLOTS for processing.

If so, when using RocksDB as the state backend, does it maintain state separately for each topic, or is the state shared across all topics processed by that TaskManager?

Flink isn't coupled to Kafka; whatever state a job needs is maintained independently from Kafka. If do you choose to use Kafka, the KafkaSource operators in your job will keep track of the offsets for each partition and store that info in the state backend, and the KafkaSink operators will similarly keep track of transaction IDs, but the rest of the job's state will not be connected in any way to Kafka.