This message was deleted.

Hi Igor. This is a good place to read about tuning. Here's the Broker documentation.

I can share since it’s POC we ingesting approx 8Bil events from kafka with rollup, keeping 24 hrs of 5 mins buckets. Schema is time bucket + 4 long fields as part of dimensions: accountId, campaignId, adId, publisherId) and then anothery 4 long fields of kpis(1-4) meanwhile it’s only 1 server. pretty strong one(40 cores, 256GB memory) everything is running there as ‘large’ setup + some memory/buffer increases for broker(e.g.) I tried to run client that takes 100 Top accountIds(computed by ordering sum(adIds) desc) and then executes under diff number of threads(10 to 100) query with PreparedStatement that samples accountId and then aggregates data to hourly granularity for this account, something like:

"select TIME_FLOOR(__time, 'PT1H') as _hour, " +
                "       SUM(kpi1) as kpi1, " +
                "       SUM(kpi2) as kpi2, " +
                "       SUM(kpi3) as kpi3, " +
                "       SUM(kpi4) as kpi4 " +
                "from raw_events " +
                "where __time >= CURRENT_TIMESTAMP - INTERVAL '24' HOUR " +
                "and accountId= ? " +
                "group by 1 " +
                "order by 1";

and then measured latency of query execution. Basically we are pleased with latency, however looking at heap of broker it’s not clear to me why it increased and continues to keep something in memory even though test finishes and full GC is triggered. So either it a)some misconfig from my side b) some missing code piece that release something or c) some leak(less probable)

Hi @Igor Berman.Take a look at heap sizing for the Broker. It points to 3 major sources of heap usage: • Partial unmerged query results.. these should not persist on heap after queries are completed. • The segment timeline and cached segment metadata do persist on heap throughout the broker's normal operation. Its size will depend on the number of segments and number of columns in the segments. How many segments did you end up with for the 4 billion rows? Another important optimization strategy is segment sizing which aims to create segments of approx 5 million rows / 300-700 MB each.

I’ve reduced retention lately to be 1/4 of what it was before. currently I have 929,631,452 total rows(after rollup), 90 segments, compaction strategy of up to 20Mil rows per segment. the Datasource is fully compacted(and was during the test), segments sizes are between 50MB(5Mil rows) and 140MB(20Mil rows)

Sounds good. Also, do you see that grow over time or it is stabilizing with a stable data set? If streaming continues, brokers also keep a record of any real-time segments being built.

it remains flat after test is done, but during test, when i go from 10 concurrent client threads to 100 concurrent threads it’s increasing

That makes sense while the queries are running. Does it stay high after the workload completes?

yes, it remains high

are you using JDBC (Avatica)?

oh! great to know. we indeed using jdbc(Avatica)!

OK, I am pretty sure you are running into that leak then. Sorry, in the meantime the best workaround is to restart your Broker, which will clear out those objects. Or, you can apply the patch in the linked PR if you are building from source

Thanks for fast response! You building great product!

glad you like it so far 😄

@Gian Merlino last question 🙂 when do you think 24.0.1 will be released? (just to understand if we can wait for it or will need to build ourselves)

Can be tough to nail down a specific day in advance, since the voting process for each RC is minimum 72 hours even for a small release, by policy. However, I would guess release would go out in the next 1-2 weeks. Once an RC is up for voting, you can monitor the dev list for progress

fair enough. I’ll monitor dev list thanks!

@Igor Berman, i was wondering if v24.0.1 fixes your problem? We are facing a similar problem with an increasing heap that remains high.

@Cl A Us yes, we see that it is better in v24.0.1. Having said that, we still see some heap problems from time to time(but they are not frequent) which version you are using?

Thanks for your quick reply. we are using v24.0.1. We migrated from v.0.22.1 and we are not sure if it got worse after the update.

I understood as well that it’s connected to sql api, if you using native api, than this memory leak(and it’s fix) is not relevant

Thanks for the hint. We are using the avatica sql driver to query for druid datasources. 😞

thanks a lot @Cl A Us for the update! will need to upgrade then, we do have occasional need for broker restarts in production so it may explain this please tell me how it’s going with 25.0.0