With exactly once enabled for my kafka producers, ...
# troubleshooting
b
With exactly once enabled for my kafka producers, I am seeing extremely long startup times in my jobs. The sink task is stuck in Initializing for 30-60mins and I see hundreds of log messages like this
Copy code
ProducerId set to 969204 with epoch 8
Invoking InitProducerId for the first time in order
Discovered transaction coordinator
ProducerId set to 970010 with epoch 8
Invoking InitProducerId for the first time in order
Why are so many producers being created, and what determines the number of producers? I would expect the number of producers to equal the parallelism (8 in this case), but that doesn't seem to be the case?
I should add that the problem only begins after the job has been running for a while and it is started from a savepoint.