https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
r
I think the cachedNamespace you can configure onHeap or offHeap bc it uses mapDb
but the Kafka Lookups is uses the underline local cache, so that depends on the cache config
d
Hi @Renato Santos where can i configure cachedNamespace to off-heap?
r
druid.lookup.namespace.cache.type
it's not on the json where you set up the lookup, it's in the cluster (Broker, Peon, Historical) config
d
thanks @Renato Santos will try that. So if i have say a lookup which is in heap and after this config changes, will it be removed and kept in offheap files?
r
yes, after the reload, they will not be stored anymore on heap
it still count as memory used, tho, but I'm not sure if it count on the jvm, or just at the kernel level (as it mmap)
https://mapdb.org/blog/mmap_files_alloc_and_jvm_crash/ i dunno the implementation ,but there's a lot of warnings and reasons that this could go wrong
d
thank you @Renato Santos for sharing the above link. I have gone through it and what i understand is JVM crash happens mainly during write and read shouldn't cause a crash.
And if offheap is enabled and when the lookups are created, it should crash during write and hence we can schedule the lookup creation during off hours right?
r
I do not use this extension, so I'm not sure, I think it re-scan periodically, but don't do always do a full-scan
I only use static lookups with a few thousand keys
if what you are doing is for joining, I recommend you to re-import the data with the lookup during the ingestion, as usually this doesn't change that much but if the lookups changes, that's one of the hard problems, you gotcha need more RAM or use another solution for the query (spark?)
2 Views
Open in Slack
PreviousNext
Powered by