This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

11/14/2022, 8:07 PM

This message was deleted.

dontu balu

11/14/2022, 9:12 PM

@Sergio Ferragut

Sergio Ferragut

11/14/2022, 9:24 PM

I'll give this a shot. I'm not an expert. Seems to me like the most likely explanation is that this is a network issue. Are you able to use curl successfully from the same server where you are running this?

dontu balu

11/14/2022, 9:30 PM

Curl always works with sending request to HTTP end point. But for some reason when druid is posting the metrics to HTTP end point, it fails with

org.eclipse.jetty.io.EofException: Early EOF

exception. Please note that it doesn't happen for all requests. For some requests, I am able to process them, and for some I get the exception.

dontu balu

11/14/2022, 9:34 PM

I feel like druid is closing the stream prematurely before it can be read by HTTP end point

dontu balu

11/14/2022, 10:58 PM

druid.emitter.http.minHttpTimeoutMillis=100

Setting this property minimized the number of

org.eclipse.jetty.io.EofException: Early EOF

exception. Looks like druid is closing the connection way too early than needed.

dontu balu

11/14/2022, 11:12 PM

Also I am not sure this is 100% sure good solution. I am still seeing few of them.

dontu balu

11/14/2022, 11:12 PM

I am still seeing few of ehse exceptions.

Sergio Ferragut

11/14/2022, 11:38 PM

If it is timeout related it is likely happening here: https://github.com/apache/druid/blob/8296123d895db7d06bc4517db5e767afb7862b83/core[…]va/org/apache/druid/java/util/emitter/core/HttpPostEmitter.java

lastBatchFillTimeMillis

is set based on

druid.emitter.http.minHttpTimeoutMillis

. Perhaps your HTTP server is processing slowly? Have you tried a much larger setting, like 500 or 1000?

4 Views

Open in Slack

Previous Next