sbj - Any idea if it's feasible to support the ...
# troubleshooting
s
Any idea if it's feasible to support the paged result control? Possibly just a dummy implementation that returns all results, and gives a control indicating no further results? I'm unfortunately dealing with some software that really don't like that it is missing...
n
You mean in the LDAP response?
s
Yes, exactly
n
s
It seems to me, looking at unboundid sdk source, that, that OID just needs to be present, with a zero length value/cookie, for it to indicate "no more results"
The product I need to hook up to lldap, seems to accept any number of results, not limiting itself to the window size it requests, but it fails because it expects that control to be present.
n
Out of curiosity, what's the product (if you can tell)?
s
If the search result included all results, disregarding the requested window size, and included that control indicating no more results, I'm pretty sure it would happily chug along
It's some horrible printer server software, for multi-office follow-me printing. From One Q/Ubiquitech, called VDMS server
n
@Firstyear do you think we could add support for that control in ldap_proto? Both in the LdapResult struct and, I guess, in the SearchRequest (or is that already the
size_limit
?)
s
Just realized ldap_proto was from kanidm. Not too familiar with it, but it does look a bit like there's some kind of support? https://github.com/kanidm/ldap3/blob/48a28b4a12de1bae6a2dabacc7ace52362fb89c3/proto/src/proto.rs#L151
n
oh, the control is from the level above the ldap op, at the ldap msg level
s
What you say sounds sensible, but i'm not very familiar with the protocol 😉
Looks like LdapControl might have some, also has that rfc referenced: https://github.com/kanidm/ldap3/blob/48a28b4a12de1bae6a2dabacc7ace52362fb89c3/proto/src/proto.rs#L91
n
yeah, it's just that I'll have to shoehorn it a bit in the design, I didn't plan on adding controls 🙂
s
Hehe, understandable 🙂
n
I'll make a simple/hacky solution for now and pray that no more controls are necessary, and if they are I'll just have to refactor and do the proper thing
s
I think a dummy control value could do it, I don't think the actual paging is really necessary
Sounds great! Thanks! 🙂
n
If you feel like contributing, I'd invite you to first of all open an issue, and then if you want to write some Rust, it should be fairly simple: https://github.com/lldap/lldap/blob/main/server/src/infra/ldap_server.rs#L47
Basically, check if the op is a SearchResultDone, and if so add the control
but yeah, create an issue anyway
s
Definitely wouldn't mind. It's been a few years since I last did any Rust though, so I might need to bother you with a couple of questions, if that's okay?
I'll get an issue created 👍
n
Of course! You can ask any question either in #975815392175980585 or in the PR
s
Thanks! I'll take a look at it. I've created the issue for now: https://github.com/lldap/lldap/issues/698
n
Great 🙂
Thanks!
s
Preliminary PR available here: https://github.com/lldap/lldap/pull/699
I need to pickup in daycare now, so I haven't had time to really test it. But it builds, and might as well get some comments already. 🙂
f
Paged search control is already there, you just need to impl the support in lldap i think
The main gotcha is keeping the "search" consistent between the two search calls, since paged search occurs over multiple calls aka multiple transactions.
n
That doesn't seem guaranteed by the RFC (looking at https://www.rfc-editor.org/rfc/rfc2696.html)
> A client may safely assume that all entries that satisfy a given search query are returned once and only once during the set of paged search requests/responses necessary to enumerate the entire result set, unless the result set for that query has changed since the searchRequest starting the request/response sequence was processed. In that case, the client may receive a given entry multiple times and/or may not receive all entries matching the given search criteria.
so it's really a best effort. As long as your results are sorted, you can run multiple independent queries with explicit pagination and it should "work"
f
Lots of clients expect it to be consistent between calls though, even if the spec allows it to diverge. Sssd is a good example here
n
Then it's a good thing that we don't implement pagination 🙂
f
Yeahhhhh.
n
but yeah, it seems like there's no easy way to implement pagination then
you'd have to store the entire query results
and then paginate on that
f
The way it's done in 389 is that you cache the query result index and then paginate on that index.
That way you have the same entry list unchanging, without needing to cache the entries
n
right, but if someone deletes a row?
Do you keep the row around and only later garbage collect it?
f
If someone deletes a row, the id in the pagination will yield an empty entry and we skip it.
And if someone adds a row, it won't be in the pagination index that was generated and we skip that too
But that works because in 389-ds we do a search and get back an IDL (ID List) which is effectively a vector of entry id numbers as 32bit ints. Then to access the entry we just grab the entry by that id.
So it's cheap to keep the IDL in the paged search.
n
Also, what I don't like is that cache is state, and I want to keep all the state in the DB
So unless I store the list results back in the DB, then query that and use that to reply...
f
Sqlite and the friends have a "row id". So you can query, get all the row ids, then use that to paged search on
And you can compress and stuff the rowids in a signed/encrypted blib in the paged search cookie
That way you dont have to cache anything
n
Oh, wow, just like that 😂 How big are your cookies?!
f
Nah thats not how 389 does it, we keep the idl in the connection struct
n
If you get to the point that you need pagination, do you want the cookie?
f
Which you could do here anyway
The cookie ids the psearch
Imagine a client launches two psearches that are interleaved
n
Oh right, I guess I have some state in the ldap connection
f
Then they poll then alternately
Yeah so you can stuff the rowids in your conn struct, and the cookie ids the psearch.
And rowids are wall smaller than whole entries
Wayyy*
s
Yeah. At least unboundid sdk interprets any non-empty cookie as indication of more results, so you could lump in whatever state you need into that. Even the remainder of the search results, that would make an easy implementation 😉 (I kid...)
f
Ldap? Easy? I need to get someof whatever you have been drinking 🤣