https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
i
Check the historical log to see if there is any indication of why it went down. It may be OOM
f
I see nothing on historical logs.
i
You should not need the priorities as the defaults place streaming ahead of batch by default. You may find more information in the syslog of the server regarding why the process was killed
f
I want change the order, first batch and second kafka. That is why I need to change the priority. As I run a druid cluster using docker-compose, I don't know hwo to get logs in syslog
i
what is the xmx for the historical?
f
Copy code
512m
i
do you see a lot of GC, especially long GC in the historical log?
I would try to increase that and see if it affects the behavior. How many historicals?
f
Here the last log I can get from the container historical before crashing. I have one historical (and changed retention rule)
Copy code
2023-10-19 14:35:33 2023-10-19T12:35:33,083 DEBUG [qtp1298547491-78] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
2023-10-19 14:35:33 2023-10-19T12:35:33,989 INFO [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment testdatasource-3_2015-01-01T00:00:00.000Z_2016-01-01T00:00:00.000Z_2023-10-19T12:35:12.369Z
2023-10-19 14:35:34 2023-10-19T12:35:34,058 INFO [SimpleDataSegmentChangeHandler-0] org.apache.druid.segment.loading.LocalDataSegmentPuller - Copied 2523 bytes from [/opt/data/segmentstestdatasource-3/2015-01-01T00:00:00.000Z_2016-01-01T00:00:00.000Z/2023-10-19T12:35:12.369Z/0/index] to [/opt/druid/var/druid/segment-cache/testdatasource-3/2015-01-01T00:00:00.000Z_2016-01-01T00:00:00.000Z/2023-10-19T12:35:12.369Z/0]
2023-10-19 14:35:34 2023-10-19T12:35:34,086 INFO [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[testdatasource-3_2015-01-01T00:00:00.000Z_2016-01-01T00:00:00.000Z_2023-10-19T12:35:12.369Z] at new path[/druid/segments/172.24.0.10:8083/172.24.0.10:8083_historical__default_tier_2023-10-19T12:35:34.083Z_3c0209abfb5a4f0792dde0412a3af6461]
2023-10-19 14:35:34 2023-10-19T12:35:34,133 DEBUG [SegmentChangeRequestHistory] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid-internal/v1/segments?counter=4&hash=1697718572055&timeout=240000 HTTP/1.1 200
2023-10-19 14:35:34 2023-10-19T12:35:34,140 DEBUG [SegmentChangeRequestHistory] org.apache.druid.jetty.RequestLog - 172.24.0.9 GET //172.24.0.10:8083/druid-internal/v1/segments?counter=4&hash=1697718572055&timeout=240000 HTTP/1.1 200
2023-10-19 14:35:34 2023-10-19T12:35:34,166 DEBUG [qtp1298547491-78] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
2023-10-19 14:35:34 2023-10-19T12:35:34,166 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.druid.jetty.RequestLog - 172.24.0.5 POST //172.24.0.10:8083/druid-internal/v1/segments/changeRequests?timeout=300000 HTTP/1.1 200
2023-10-19 14:35:35 2023-10-19T12:35:35,198 DEBUG [qtp1298547491-73] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
2023-10-19 14:35:36 2023-10-19T12:35:36,237 DEBUG [qtp1298547491-87] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
2023-10-19 14:35:37 2023-10-19T12:35:37,267 DEBUG [qtp1298547491-73] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
2023-10-19 14:35:38 2023-10-19T12:35:38,311 DEBUG [qtp1298547491-87] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
2023-10-19 14:35:40 2023-10-19T12:35:39,252 DEBUG [qtp1298547491-73] org.apache.druid.jetty.RequestLog - 172.24.0.9 POST //172.24.0.10:8083/druid/v2/ HTTP/1.1 204
2023-10-19 14:35:40 2023-10-19T12:35:39,366 DEBUG [qtp1298547491-87] org.apache.druid.jetty.RequestLog - 172.24.0.5 GET //172.24.0.10:8083/druid/listen/v1/lookups HTTP/1.1 200
I tried increasing xmx and it changes nothing
i
can you post full log?
do you have lookups configured?
f
Yes I have lookup configured. Here the full log.
full-historical-log.txt
And the log of index_parallel task
log-index-parallel.log
And I see this kind of warning in the log of the index_kafka task
Copy code
2023-10-19T12:37:20,880 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [3] Segments
i
I do not see anything in the log provided thus far but if I had to guess, historical is OOMing. You may want to try to tail the historical log when this happens to catch it. It has to handle the segments AND the lookups in memory
f
ok thanks Ian. I will try.
i
You could also try the quickstart from Imply downloads to see if this works there. That may be a better test cluster to try.
f
Good idea!
a
forceTimeChunkLock - false uses segment locking instead of timechunk locking. Is this intentional?
f
yes it is intentional
in order to synchronize index_parallel and index_kafka
a
Is appendToExisting set to true for the index_parallel job?
f
No
a
I see, thanks for sharing!
f
Just to add more information on the strange behavior I'm having, after restarting the history node, the broker crashed, and after restarting the broker, my index_kafka finished fine.
Open in Slack
PreviousNext
Powered by