:wave: Hello everyone, My team is using a `RichAs...
# troubleshooting
c
πŸ‘‹ Hello everyone, My team is using a
RichAsyncFunction
to call an external HTTP service, with
org.apache.http.nio.client.HttpAsyncClient
as HTTP client. It is an async HTTP client that follows the non-blocking I/O (NIO) model, allowing it to handle many concurrent HTTP connections with a very small number of threads, using the Reactor pattern. I'm trying to figure out the best way to create and configure this HTTP client when used inside a RichAsyncFunction: 1. Client creation: Is it best to create one client for each subtask (in the
open()
method), or should we share a single static client across all operator subtasks in the same JVM? I've read this last option is an anti-pattern in Flink, but just wanted to see if it still makes sense. 2. Num threads: How should we configure the number of IO dispatch threads in the client? Should we limit it to one thread per client and rely on Flink's parallelism, or set it higher? By default this library configures 2 threads. Should we set as many threads as CPU processors available to the JVM (
Runtime.getRuntime.availableProcessors()
)? Cheers!
j
should we share a single static client across all operator subtasks in the same JVM? I've read this last option is an anti-pattern in Flink, but just wanted to see if it still makes sense.
Where did you find this as an anti-pattern? Could you please share the source for this?