Hi team, there is a performance issue when using ‘JSONPATH’ transformation functions, this is may be caused by the jayway Inefficient LRUCache design. When encountering high concurrency in data ingestion, there will be fierce competition for CPU and all ingestion threads are waiting for the lock, and data consumption will be delay.

Hi @wentao jin, good find 👍 In order to confirm your suspicion that the problem is due to contention, it would help to disable the cache entirely. I'm new to pinot, but I took a look at the jayway library and it looks like you can override the cache implementation by calling

com.jayway.jsonpath.spi.cache.CacheProvider.setCache(new NOOPCache());

but you need to do it before anything else in the same

ClassLoader

calls

CacheProvider.getCache

to make sure the registration works. To really confirm this is related to contention, it would be great to get a contention profile from async-profiler without overriding the cache provider. You can run it as a native agent as outlined here, and use this JVM setting:

-agentpath:/path/to/libasyncProfiler.so=start,event=lock,file=contention.html

(Or set

-agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=cpu.html

to get a cpu profile which should identify the bottleneck without proving the issue is contention) Getting a look at the flamegraph will really help to understand whether this is contention or just slow JSON path evaluation.

Hi @Richard Startin. I have run the async-profiler tool to analyze CPU and lock.

Pinot Server Flavor: 28vcpu.110mem.2000ssd

"transformConfigs": [
  {
    "columnName": "correlationId",
    "transformFunction": "jsonPathString(report,'$.identifiers.correlationId','')"
  }
...
]

I have done several times profiling, each time lasts 1 minute. The result is basically the same. Lock wait of LRUCache and CPU usage of JsonPath are both high.

I wasn't suggesting the NOOPCache as a solution, but as a diagnostic step

Sure

oh sorry, I meant can I have the actual files, not screenshots?

cpu-analysis.html,lock-analysis.html,noopcache-cpu-analysis.html,noopcache-lock-analysis.html

yep, this is mostly the LRUCache indeed

Prometheus JMX client performance is bad too, It will get better after adding whitelistObjectNames configure.

In the short term, I suggest you keep the

ConcurrentHashMap

cache you have implemented if it solves your problem, but I'm not sure it will be suitable for every pinot user if it caches every jsonpath. Since pinot has a guava dependency, we could instead use guava's cache with a sensible size limit and register it to avoid jayway's own LRUCache. If you're interested, it could be a nice open source contribution to propose.

I’m thinking about this. Do we really need LRU Cache in this scenario. In fact, HashMap still has very high query performance in the order of 100,000, and for big data scenarios, this does not take up a lot of JVM memory, and basically It is impossible to have more than so many json path transformation configurations. In the case of frequent data ingestion, Even if it exceeds 100000(may other sensible size), Not cache json path may be better than frequent swapping in and out of LRU.

I think you might be right. WDYT @Mayank?

Yeah, seems reasonable.

Hi @wentao jin I created an issue here, you can link to it if you want to contribute your change 👍 https://github.com/apache/pinot/issues/7403

Thanks Richard.