Is there documentation somewhere about developing ...
# lucee
s
Is there documentation somewhere about developing cache extensions and the cache interface requirements? I'm trying to troubleshoot some issues with using the MongoDB extension as session storage, and hoping to improve / optimize the cache extension (most of it was released as beta by Andrea Capolonghi about 7 years ago and I would like to "finish it").
b
@seandaniels I'm not sure if there is docs to be honest
But I've written some cache extensions for ortus
I basically looked at some of the existing open source libraries out there
s
Yes, I was gonna ask you next if you'd let me peek at your MongoDB cache code šŸ™‚
I know that's a paid Ortus product though. So no worries.
b
The main bits are • the Driver CFC that teaches the admin UI about the cache • the java class that implements Lucee's
Cache
interface Then you wrap it up in an extension with the right stuff in the right folders and it will install.
Yeah, our mongoDB extension is a closed-source commercial product, but there are several LAS extensinos on Github
We don't necessarily do our builds the same way they do-- so long as the lex file you produce is in the right format, that's all that matter.
s
Yeah. The LAS mongo extension is the one I'm talking about. I've been "maintaining" it for the past few years. But the cache part of it was always beta and seems super non-optimal. But there are a lot of functions I'm just not sure why they are there.
b
The biggest reason we created our own Ortus extension for stuff like Mongo, Redis Couchbase, etc was that the LAS-maintained extensions weren't really 100% there and had no support really.
We needed to be able to use stuff with our clients that we knew we could fix.
s
Heh. Yup, ok.
b
I can help answer questions about the Cache interface as I've spent a lot of time over he years bugging Micha about it.
s
The real big black box to me is how the session storage interacts with the cache. I'm seeing way more database chatter per request than I would expect.
b
but I can't really speak to any particular code already in the mongo extension as I'm not really familiar with it
That question isn't really even about the cache design, that's just how Lucee does sessions.
s
i.e. sometimes it looks like it calls getEntry() about 4 times at the beginning of a request. And it always overwrites with cache entry at the very end with a new one (hits = 0)
Right
b
There will probably be more chatter than you expect
I THINK Lucee does it per key now to try and prevent race conditions
I'd start by putting in some debugging to log what is got and set for a single request
s
I guess that's what I'm hoping to understand. How does the Lucee session interact with the cache. There is some very slim documentation about using a datasource as session store, but I'm not sure if that directly applies.
b
It's difficult to debug directly on the external cache side as the objects stored are serialized Java objects that contain the actual data
The
sessionCluster
flag controls whether Lucee trusts the cache only as the primary source of data.
Lucee will store the session data in a local in-memory cache for a short amount of time, and if session cluster is off, it will favor what's in memory and assume the cache hasn't changed.
s
I have FusionReactor and can see in the "Relations" tab for a request all of the reads and writes to MongoDB. That's pretty helpful but that's where I'm seeing a lot of extra calls I would not expect. I have
sessionCluster=true
.
b
Back when I was first writing the couchbase extension, which was on Railo 4.x it was usually one get at the start of the request and one set at the end, but I think Micha stores each session key separately now
s
The datasource as session store docs seemed to indicate when sessionCluster is true the session data would be read at the beginning of a request and held in memory until then written back at the end of request. But I'm seeing dozens of reads/writes to the session collection per request.
Huh. Interesting.
b
Those docs were probably never updated, lol
It's been quite a while since I've looked at the code.
s
So each key in the session scope I try to read in a request would be a separate call out to the cache? Yikes. Hm.
b
I think the best mileage still is to put some logging in the mongo source code that tracks the keys and values being set.
šŸ‘ 1
I'm not 100% sure about that, but I'm thinking he did that a while back
The problem was two requests runing at the same time that modified two separate keys in the session scope would overwrite each other
Also, setting a small key in the session scope had extra overhead if you had other large keys that didn't actually change
So Micha was trying to optimize it to just read and write what changed, with the trade off of more traffic across the wire
But I'm not aware of any documentation whatsoever that explains this behavior
I may be able to find some JIRA ticketes where he discussed it
s
OK. That makes sense. The keys in my session store though don't look like they would support that. They are format:
lucee-storage:session:2d9cfadc-456e-4586-abdb-f6b966d34b7a:dealstream-com
where the UUID there is the CFID value. Maybe that's a problem with the cache extension though.
b
I doubt you'll find a useful ticket that represents the addition. It's probably just committed under a generic ticket labeled "improve caching" šŸ˜†
šŸ˜‚ 1
• lucee-storage generic prefix for storage • session as opposed to client • 2d9cfadc-456e-4586-abdb-f6b966d34b7a This is likely session ID like cfid • dealstream-com This is probably your application name, but it could be the key. You may recognize it
s
OK. Well, thanks for the sanity check anyway. If you do happen to recall anything that talks about this I'd be psyched to see anything that sheds light on it. Otherwise yeah I'll start adding some log/debug stuff.
Yes, application name
b
hmm, well that doesn't support the theory of having a separate item for each key, but he was talking about doing that after I had developed the couchbase extension and I never really dug back into the code at a deep level after that
šŸ‘ 1
There's a longish comment thread on this ticket which is where I got most of that idea
šŸ‘ 1
Looks like it only applied to
this.sessionCluster = false
s
Yep
b
Just re-read the comment thread on that one. Geez, just a disappointing and ridiculously unnecessary discussion šŸ˜•
😩 1
s
Interesting. By the way, this app does use the
sessionStorage@cbstorages
module exclusively for session data so your comment about that in the ticket is probably quite relevant
z
Micha did ask for feedback on what improvements you'd like for the cache interfaces on the mailing list, but @bdw429s you didn't respond yet. We are chatting in a temporal black hole here, post to the mailing list!
b
I'm aware and it's on my list of things to look at.
I need to review the interfaces again to recall everything I wanted changed, which was about 7 years ago at this point šŸ™‚
s
Zac, thanks I will start a thread or try to join an existing thread there as well.
b
Sadly, most of that conversation happened years ago over Skype with Micha!
To be clear, the conversation here really has nothing to do with the
Cache
interface, nor caches at all. it's with how Lucee treats session storage.
The caches themselves are simply black boxes with get and set method (more less) that Luce taps as-necessary.