https://venicedb.org logo
Join Slack
Powered by
For folks who might be interested (and by that, I ...
# general
z
For folks who might be interested (and by that, I mean venice devs), I have added a TON of new functionality to the heap dump parsing tool me and @Kai-Sern Lim made. It's now pretty fast and tracks all Java objects I've used. It's also got a very interesting property. If you parse the heapdump to a directory of parquet files, your favorite local LLM is pretty good at querying it with pyarrow. https://github.com/ZacAttack/HeapDumpStarDiver
🙌 2
❤️ 2
I need to jot down a few recipes (similar to the OQL snippets wiki I maintained). But I think this might be a pretty big quality of life improvement
You'd probably have fun with this @Manu. I was able to load the heap dump into DuckDB and put an LLM in front of it and start asking it questions about objects on the heap. I was able on a prompt just say "What's in this hashmap?" and it was able to print out what was in the hashmap and show the duckDB query to get it.
🚀 1
e
Nice!
f
Cool! Did you pitch it to JavaZone?
z
I did, submitted the talk last night
🚀 1
s
I would like to try it out. So the only requirement to use it is to have rust installed?
z
Yeah, rust and the other project dependency mentioned in the readme
s
Got it. Thanks!
z
Alright. I might be beating this horse, but this has been a lot of fun. I've gotten the tool to be even faster, and very interestingly, I've added a flag called --robo-mode. I'll need to do a write up of robo-mode later, but the gist of robo-mode is that it manages to speed up the parsing of the hprof quite a lot, however, it results in a less human friendly format for the parquet files, but it IS a more friendly LLM format. So if your goal is to just ingest this data into some context and then query it with an LLM, robo-mode seems to be a straight win. Though some caution, I was showing this to a friend and he asked me how I knew the bot was correct, and I realized I had no idea, nor any easy way to validate the finding because the new data format is very tedious for humans to deal with 😂 😅 . But it's a very interesting avenue.
🚀 1
💪 1
f
E2E DIV should now include LLM hops? Talk about scope creep 😅
😮 1
😅 1
s
I've bunch of heaps to analyze. I'll try out the latest changes.
Sleek
Copy code
HeapDumpStarDiver completed in 1m 44s for the 6.7GB dump. Here's the profiling breakdown:

  ┌────────────────────────────┬────────┬───────────────────────────────────────────────────────────┐
  │           Phase            │  Time  │                          Details                          │
  ├────────────────────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ Phase 1a (sequential scan) │ 1.3s   │ 409,363 utf8 strings, 31,026 classes, 7,619 segments      │
  ├────────────────────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ Phase 1b (parallel index)  │ 34.2s  │ 83,442,503 object mappings, 10,693,785 primitive mappings │
  ├────────────────────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ Phase 1c (finalize)        │ 0.0s   │ —                                                         │
  ├────────────────────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ Pass 2 (write parquet)     │ 60.0s  │ 31,026 schemas                                            │
  ├────────────────────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ Writers close              │ 8.0s   │ Flush to disk                                             │
  ├────────────────────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ Total wall clock           │ 1m 44s │ 321% CPU utilization (good parallelism)                   │
  └────────────────────────────┴────────┴───────────────────────────────────────────────────────────┘
Copy code
9,654 MetricsRepository instances — that's a lot! Each likely holds many sensors. And 203,651 SensorMBeans. Let me now compute a comprehensive summary.
🤯
z
If you could let me know how much slower visualvm is on that, that'd be a great data point for my talk 😂
Also an experiment I wanna try. Explain to it what a PCS is and have it print it across ingestion tasks
s
It finished the the analysis in less than 5mins. (Visual VM takes more than 5mins just to open the heap)
Also an experiment I wanna try. Explain to it what a PCS is and have it print it across ingestion tasks
haha. I've been using it for that (we had a bug where leaderTopic was messed up during blob transfer.. it did great job figuring it out 😄)
z
Sick!!!
s
Copy code
Key Ratios (per ingestion task)

  ┌───────────────────────────┬─────────────┬─────────────────────────────────────┐
  │          Metric           │    Value    │             Assessment              │
  ├───────────────────────────┼─────────────┼─────────────────────────────────────┤
  │ MetricsRepos per task     │ 19.7        │ High — each creates sensors, MBeans │
  ├───────────────────────────┼─────────────┼─────────────────────────────────────┤
  │ SensorMBeans per task     │ 416         │ Very high                           │
  ├───────────────────────────┼─────────────┼─────────────────────────────────────┤
  │ Tehuti Sensors per task   │ 894         │ Very high                           │
  ├───────────────────────────┼─────────────┼─────────────────────────────────────┤
  │ PartitionConsumptionState │ 1,355 total │ Reasonable for 490 tasks            │
  └───────────────────────────┴─────────────┴─────────────────────────────────────┘
z
Also interesting note. It also took ~40s for my 20GB heap dump.... Which tells me there's another bottle neck in there. So more speed gains in there somewhere!
s
How did you speed up initial hprof parsing?
LLM + HeapDumpStarDiver
❤️
Copy code
⏺ Now let me check the key insight — how many of the 1,491 topics does each consumer actually subscribe to vs how many it caches.
z
Lots of things. Mostly with parallelism. The hprof file format is written in chunks. So just spawn threads for all the chunks. Also use drainer threads (like our SIT), and use seperate files so no contension
🚀 1
Also your change introduced me to Rayon
Which is very good at this it turns out
Idk why visualvm doesn't leverage the chunks....
😂 1
s
It's so fast! The following analysis could have taken at least 6H+
❤️ 1
🤖 1
sample-analysis.txt
sample-analysis.txt
z
There's a blog series that dives this for hprof slurp. That also helped
s
Could you please share the link?
z
It's a 3 part series
s
Added some scripts/skills in our internal 
VeniceBots
to do JXRay style analysis. If this is useful, I can cut a PR to merge them in the https://github.com/ZacAttack/HeapDumpStarDiver
Copy code
Here are the results for this 6.7 GB Venice server heap dump (94M objects, 42K classes, robo mode):

  JXRay Waste Summary — 1.8 GB estimated waste

  ┌─────┬───────────────────────────┬──────────┬────────┬──────────┐
  │  #  │           Check           │ Affected │ Waste  │ Severity │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 1   │ Bad Collections           │ 3.7M     │ 408 MB │ CRITICAL │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 2   │ Duplicate byte[]          │ 7.3M     │ 363 MB │ CRITICAL │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 3   │ Class Count               │ 42,396   │ 331 MB │ MEDIUM   │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 4   │ Duplicate Strings         │ 4.6M     │ 275 MB │ CRITICAL │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 5   │ Bad Primitive Arrays      │ 610K     │ 208 MB │ CRITICAL │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 6   │ DirectByteBuffer Off-Heap │ 9,448    │ 169 MB │ CRITICAL │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 7   │ Collection Sizing         │ 834K     │ 33 MB  │ HIGH     │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 8   │ Boxed Primitives          │ 1.5M     │ 24 MB  │ HIGH     │
  ├─────┼───────────────────────────┼──────────┼────────┼──────────┤
  │ 9   │ GC Roots                  │ 43,464   │ —      │ INFO     │
  └─────┴───────────────────────────┴──────────┴────────┴──────────┘

  Key findings for this Venice server:

  1. Bad Collections (408 MB) — The biggest waste. 672K empty HashMaps + 657K single-element HashMaps, 1M empty ArrayLists + 1M single-element ArrayLists. This is likely driven by per-partition/per-store metadata objects that
  initialize collections they rarely populate.
  2. Duplicate byte[] (363 MB) — 7.3M duplicate arrays across only 500K unique values. Likely serialized partition metadata, topic names, or Avro schemas being duplicated per consumer/producer.
  3. Duplicate Strings (275 MB) — 4.6M dupes, worst single string duplicated 377K times (auto-sampled at 20% due to 7M+ strings). Store names and topic names repeated per partition are the usual culprit.
  4. Bad Primitive Arrays (208 MB) — 400K wasteful byte[], 124K wasteful int[], 81K wasteful long[]. The float[] waste (93 MB) stands out — likely Tehuti/metrics related pre-allocated but unused sensor arrays.
  5. 42K classes loaded — classloader concern territory. Many are lambda classes (note the $$Lambda$ proliferation in ingestion tasks — 80+ lambda class definitions just for AA/LF tasks).
  6. Venice-specific context — 1,140 MBeans per ingestion task and 1,786 Kafka metadata objects per task confirms known metrics/metadata explosion. 2.16M TopicPartition objects is a major contributor to duplicated strings and byte
  arrays.
follow-up-correlation-of-jxray-with-code.txt
follow-up-correlation-of-jxray-with-code.txt
z
Please do!
s
Ack. Will cut PR later today. Thanks!
Created PRs last night. Let me know if they are useful. Thanks!
z
three PR's!
The frame one is interesting. So this is now some custom formatting for frames? Also the OOM thread detection is super cool!
s
I think it's a direct 1:1 mapping of what's already in the HPROF file. (was it already there? )
z
Yeah they're on there. I think the queries are cool. I need to check how stack frames were being handled before this, but I'm willing to bet your change makes them more obviously parseable. I'll give it a shot when I get back to my home computer tonight
s
Yeah sure. Thanks
3 Views
Open in Slack
PreviousNext
Powered by