Anyone have any good strategies for troubleshooting issues with distributed cache when using ehCache? We do not serialize objects in the cache, but just have ehCache announce the expiration. We have found this offers much better performance, since there's no heavy serialization going on. However, we've seen issue both in ACF & Lucee where occasionally a node's cache is showing outdated information. I believe there's an issue in the network where the announcement event never gets received by node. This is not a frequent problem, but happens enough that it causes issues because users start seeing outdated information. I've never been able to replicate the behavior, which is why I think the problem may be in the network of our production environment, but it could be I'm just not testing enough load to see the behavior.

So are you saying that since you utilize notification of the expiration that you then perform a manual action (like eviction) but that is not occurring? Or you are letting the expiration take place in the lazy manner automatically but its not working?

We expire the object programmatically (via

cacheRemove()

(which is done via CacheBox). We have ehCache set to just announce evictions. This works correctly like 99.9% of the time, but occasionally a node gets out of wack and an eviction was not seen.

Well with replication vs distributed could be a bottleneck; also depends on size of your cache since replication does a complete copy of the cache to each node and requires a very effective network...so to me it sounds like a bottleneck in your network or some sort of hiccup. esp if you are saying literally 99.9% is good thats almost worth ignoring at that certainty rate

None of our replicate is actually using serialization. All of the "OnCopy" options are false, so it should only be announcing events to evict items from cache.

Give it a try, I havent utilized RMI. Also curious if its issue with app (cachebox) receiving or is it ehcache not notifying...do you have logging implemented on both sides to see who flops

I can't imagine the issue is w/CacheBox, because it's just calling the native CF cache functions.

Patrick -- is this in the cloud or on prem? Do you have access to do a wireshark or vmware port mirroring to see if you are getting those eviction notifications?

It's on a private cloud in a datacenter, so we can run network inspections if need. Any recommendations on what exact traffic I should be monitoring to look for failures in RMI events?

I know you said it didn't happen regularly, but sometimes it's easiest to capture all data between the caching server and your ACF server. Then pull it into Wireshark for the complete picture. If you don't have enough room to store everything, you can pick just the RMI protocols, but there is a chance you could still miss it in that case. Next time you have a bad cache hit, look at the wireshark trace from that time and see if you actually got the release-cache packet from the ehCache server. It should be pretty easy to pick out if it is there. That will at least eliminate any config issues on the ehCache server and the network.

@quetwo Thanks for the ideas!

Are there any time-to-live settings you could tweak? I haven't used Ehcache for a while but, from what I remember, the default time-to-live was then 0 (meaning: infinite lifetime).

We already set our TTL is set to 30 minutes. It's certainly possible that the cache expiration is failing more often then we get reports on, because a user would only see it in a corner case scenario. This application still uses sticky sessions, so a user should stick to the same server unless something unexpected happens. So a user will only notice the problem is they are on a node which has loaded the entity within the past 30 minutes and the expiration event fails to replicate. I don't think that scenario happens with high frequency because most people who are making changes to an entity are likely the only people making a change to it.

What you have observed might not be failure. It just might be the way Ehcache is meant to work. We've discussed time-to-live(TTL). However, there is another Ehcache setting that can affect cache expiration, namely,

memoryStoreEvictionPolicy

Possible values for memoryStoreEvictionPolicy are

LRU

(Least Recently Used),

LFU

(Least Frequently Used), and

FIFO

(First In First Out ). The documentation suggests that these policies are all probabilistic. In other words, eviction is guaranteed to occur 99% of the time, but not 100% of the time. The default eviction policy is

LRU

. To this, the documentation adds, "_If probabilistic eviction does not suit your application, a true Least Recently Used deterministic algorithm is available by setting_ `java -Dnet.sf.ehcache.use.classic.lru=true`" https://www.ehcache.org/documentation/2.8/apis/cache-eviction-algorithms.html

@bkbk Thanks for the input. In our case we're specifically triggering an expiration which should force the event to be honored by all the replicated nodes.