https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
b
@Shilpa Sivanesan Please check the order of rules applied, if you have loadbyforever alongwith loadbyinterval, then loadforever should take precedency.
s
@Bibek Sahoo the loadforever is applied after loadbyinterval
b
@Shilpa Sivanesan I believe you should not have LOADFOREVER rule set as you want only to load by interval. https://druid.apache.org/docs/latest/operations/rule-configuration.html#rule-structure
The segements that satisfied rule1 will also satisfy rule 4 and hence will be loaded
s
I added load forever because apart from the specific intervals everything has to go to cold tier
Also these rules were working for multiple years After we switch from tranquility to native kinesis ingestion (supervisor) facing this issue
Am I missing anything specific to supervisor ?
b
supervisor should not have anything to with load rules
may be coordinator log will have some more details regarding the load rules
s
The coordinator get rules API returns the latest rules, but it doesnt gets triggered
v
@Bibek Sahoo here are coordinater logs. Does Throttling replication cause this issue? For the dates, which are supposed to move to Hot tier but are still in Cold tier, We're getting Throttling replication as below:
Copy code
2023-01-20T12:48:51,439 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Throttling replication for segment [clickstream_2021-10-17T00:00:00.000Z_2021-10-18T00:00:00.000Z_2022-01-26T09:13:58.542Z_4] in tier [hot]
2023-01-20T12:48:51,439 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Throttling replication for segment [clickstream_2021-10-17T00:00:00.000Z_2021-10-18T00:00:00.000Z_2022-01-26T09:13:58.342Z_5] in tier [hot]
For the dates, which are loaded in Hot tier as expectedly, we have't seen these Throtting replication they're getting dropped to Hot tier as expected:
Copy code
2023-01-20T01:33:29,131 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Dropping segment [clickstream_2021-10-18T00:00:00.000Z_2021-10-19T00:00:00.000Z_2022-01-27T09:15:12.255Z_2] on server [historical-hot-server-1] in tier [hot]
2023-01-20T04:18:34,666 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Dropping segment [clickstream_2021-10-18T00:00:00.000Z_2021-10-19T00:00:00.000Z_2022-01-27T09:15:12.255Z_5] on server [historical-hot-server-7] in tier [hot]
Does changing 
replicationThrottleLimit
help to resolve this? We're using 
replicationThrottleLimit=10
.
Open in Slack
PreviousNext
Powered by