This message was deleted.

best is hard, "it depends" "Classic" or "can't go wrong" is nginx => [your team tech stack] => [kafka|redpanda] => druid if your infra is all on AWS, and fears kafka nginx => [your team tech stack] => Amazon Kinesis => druid

@Renato Santos Thank you for the reply. The "classic" is what I've heard, but I guess I was struggling to understand why step [your team tech stack] is needed? I thought that the HTTP endpoint to nginx access.log would be enough? The URL endpoints are only for the purpose of collecting data and do not need to trigger additional processes. With the above in mind, why is it the 'classic' to put something like a Python API between nginx and logstash? Wouldn't this cause the data processing to slow down? Also, thanks again, I realize this pre-processing is a bit off topic for druid.

you can try to use logstash to kafka but usually you need an application anyway, unless you will always just capture clicks and return a empty gif with no cookies, Using logstash can see a little fragile in my opinion, as it intended to read from files logs, but that can work, just make sure the log-rotate is not truncating this file and duplicaitng or losing some during, and how logstash will handle this Usually for logging the expectations is not high, and a little bit of loss is or duplication is not considered the end of the word, in comparison to an API that handled everything to kafka and returned 200

@Renato Santos Just wanted to thank you again. You helped me get one step closer to being able to use druid!