This message was deleted.
# generals
Slackbot
06/22/2023, 7:26 AMThis message was deleted.
l
Luiz Augusto
06/22/2023, 9:38 AMDid you build this ingestion via web console? If yes, could you see your data in the sample?
b
Bibek Sahoo
06/23/2023, 5:03 AMDo you have number of partition:number of task as same ratio. like 1:1, If you do not then kafka might be emitting faster than druid is consuming because of task count less than number of partition in kafka topic
w
William Nelson
06/23/2023, 5:17 AMWhen you say "not getting the latest data" do you mean that it's ingesting it slowly but the data takes a long time to get in? Or do you mean that nothing is getting in at all?
If it's slow, you might take a look at your Kafka lag in the supervisor stats and see if that's growing.
Also, if your timestamps aren't right, it can sometimes look like data is coming in late. I've seen a misconfigured server send timestamps in the past, so everything was late. Compare the Kafka metadata timestamps to the timestamps in the data to see if they're close.
Also, take a look at the data you're bringing in. If it's flat structures with a few values, it should zoom.
If each Kafka message is several megabytes in size, then maybe you need to rethink some design decisions.
William Nelson
06/23/2023, 5:20 AMIf you're getting nothing at all, then you'll need to check your logs. One thing that can cause this is if the offset that Druid wants to use has been deleted by Kafka's retention policy. Druid can get stuck in a cycle of trying to get data that's already gone. In that case, you might need to reset your supervisor. Be really careful with that though, it's very easy to lose or double your data.