http://coldfusion.com logo
Join Slack
Powered by
Im seeing the following in FR which I think is Luc...
# lucee
d
Im seeing the following in FR which I think is Luce waiting on a pool connection for Redis. Is it likely that the maxConnections setting for the Redis extension is too low? Im also seeing quite high CPU usage.
Ive just set session back to memory and this has stopped, so its definitely Redis related...now just need to find out why...
g
Short of someone giving you an answer - (I have no useable Redis experience, sorry)... But perhaps if you run some experiments (does extending the connecting count work? / does extending (or reducing) the timeout work? And update here, as you go - someone (else / more knowledgeable ) might be able to chime-in with some nuanced advice?
d
That's definitely the plan for today, though its not easy to run experiments in production in our environment. This same configuration has been working on our nonprod envs for weeks now, but obviously that doesnt get the same use as prod. I am fairly sure Ive got maxTotal set way too low in prod for the number of users we have, so my plan today is to determine the usage of sessions across our production cluster work out how that would relate to the Redis pool config. Also add some custom metrics to FR to show how the app is using the pool.
For interests sake, Ive just worked out we have around 21k active sessions across our cluster. And its not peak load time yet. 😬
😲 1
b
I don't see any indication that it was waiting on a pool connection.
I mean, it might have been, but the profiler doesn't show that
It was this line of code here it was waiting on https://github.com/lucee/Lucee/blob/6.2/core/src/main/java/lucee/runtime/type/scope/storage/IKHandlerCache.java#L60 Which is simply a synchronized block specific to the cache key
It appears Lucee locks all operations at the key level so two threads can't be doing something with the same key at the same time.
In order to know what it was waiting on, you'd need to have captured a full thread dump of the JVM at the same time to find what thread was holding the lock on the same key and see what it was doing.
There are several cache operations inside that lock. Perhaps one of them was waiting on a connection from the pool. Or perhaps it was network latency, or a large item being deserialized, etc.
we have around 21k active sessions across our cluster
I'd recommend doing some basic bot detection and setting the session timeout to 0 for bot hits. Also consider lowering your session timeout if you think you have a large amount of transient sessions (just a couple hits and they're done)
d
Thanks Brad. These arent bots, we block those at Cloudflare. Its possible there are some API call sessions hanging around that dont need to be.
b
Ideally an API shouldn't enable session scope at all 🙂
d
Looks like our Taffy endpoints are not to blame, they all show 0 sessions created
👍 1
In sessions in FR, I can see one row per 'application', which seems to be every URL we host, these have sane values, i.e. number of active users. But there is one that has our 
this.name
from 
Application.cfc
which right now on a single server has 2500 sessions active
Hmm...I think this is our SOAP API 😐
Open in Slack
PreviousNext
Powered by