https://venicedb.org logo
Join Slack
Powered by
Hey folks, is there a working example of fast clie...
# general
d
Hey folks, is there a working example of fast client wiring? The closest thing I found to a guide is 
com.linkedin.venice.fastclient.utils.TestClientSimulator
. Is it an appropriate thing to follow (minus the mocks and dummies 🙂 )? Thanks!
f
@Xun Yin (or @xunyn8?), what's our best open source demo of how to use FC? We should probably add a page to the docs for spinning it up?
d
Hi fellas, sorry for bothering you all, just bumping this one up 🙂 . For a bit more context, we're using thin client in our Voldemort -> Venice migration path, but we're also very much interested in replacing it with the fast client. I'm back on it this week, which means back to understanding the internals of 
ClientConfig
. I'd appreciate some tips on how to initialize it correctly (reading the code and some trial and error can identify required parameters, but it doesn't remove some ambiguities, e.g. the faith of R2 client). Thanks again 🙂
f
Sorry for letting this thread stay unanswered...! I guess we don't have docs on this yet (but we definitely should...). In terms of looking at code samples, I would gravitate towards integration tests, such as: https://github.com/linkedin/venice/blob/main/internal/venice-test-common/src/integrationTest/java/com/linkedin/venice/fastclient/AvroStoreClientEndToEndTest.java
✅ 1
See the abstract class this test extends for more info... for example, we typically turn on server-side quotas in the store config of stores that will be accessed via FC (by default the quotas are enforced only on the router-side, but since FC bypasses the router, they can't come into play...). https://github.com/linkedin/venice/blob/main/internal/venice-test-common/src/integrationTest/java/com/linkedin/venice/fastclient/utils/AbstractClientEndToEndSetup.java#L226 This is probably a minor detail, as maybe you're not setting quotas at all anyway.
✅ 1
There are many permutations of the 
ClientConfig
exercised in this test, which you may be able to draw inspiration from...
d
Thanks a lot!
f
Np! Thanks for persevering with us even though we don't have shiny docs nor evangelists nor a marketing team! I promise the code is much better than the docs 🤣
😄 1
Please let us know if you have any issues or followups
✅ 1
d
Will do, thank you 🙂
🤝 1
Hi again, is there something extra to be done on set-up / infrastructure side when faced with failures to discover d2 service:
Copy code
com.linkedin.venice.client.exceptions.ServiceDiscoveryException: Failed to find d2 service for test-store after 10 attempts
Copy code
Caused by: com.linkedin.d2.balancer.ServiceUnavailableException: ServiceUnavailableException [_reason=PEGA_1012. no service properties in
 lb state, _serviceName=venice-discovery]
I have created the 
test-store
on top of the default single DC docker compose setup. Client creation is mostly copy-paste from the tests linked above (mostly == scala port, code layout and constants are pretty much the same).
f
So, for d2 service discovery, we need to have the router announce to d2, which is done via Zookeeper... There are various configs that need to be set for the right things to happen. TBH, it might be overkill to use D2 in the beginning… Just curious, do you already have some other solution in your stack for service disco? (like Consul or whatever) Or none yet?
d
Not really. I understood d2 is required for fast client, but I guess that's not the case 🤔. Although, not specifying it just leads to failures, so my understanding might be correct 🤷‍♂️
So, just to confirm: • the default single dc compose does not contain sufficient D2 config (based on the errors above) • it could be possible to use the fact client without the D2 (right now such wiring fails as they're required for request-based metadata) Is this correct? If so, I guess I'll need to study the test samples provided above a bit more...
f
Actually, I double checked with @Gaojie Liu and he told me that d2 is indeed required for FC at this time. So you had it right all along.
✅ 1
We ought to fix the docker compose to configure d2 correctly out of the box
✅ 1
Sorry for my confusion. Within LI we do use d2 and I end up forgetting when is it optional versus required… Ideally, I would prefer it to be optional everywhere, but evidently we’re not there yet.
d
If you can give me some pointers (e.g. reference config), I can play with compose. Right now I'm trying to make it work, and we will need it anyhow outside of the compose for our tests 🙂 . Regarding d2, don't worry. By going through the code, it looked to me like you're planning to make it flexible, but it's not there yet. We are fine with having it for now.
f
Circling back on this... @Xun Yin / @xunyn8 (which one is it, Xun 😅 ?) may be able to give some more detailed instructions later on how to setup D2. Sorry, everyone's been super busy, thanks for your patience. Besides, we discussed that we can probably remove the requirement from FC for D2 entirely (i.e., make it optional instead). But not sure what the level of effort is for that change. I'm going on vacation for a couple of weeks so probably won't be active during that time, but hopefully someone else from the community can get back to you on that, before I'm back.
✅ 1
d
Awesome, thank you. If I can help out somehow (for start, setting up the d2 in compose), please let me know / point me to the right direction 🙂
f
I think the easiest thing to unblock progress would be to make d2 optional in the fast client, just like it's optional in the thin client and controller client.
It probably should have been this way from the start but we just took a shortcut since within LI we do use d2, so just assumed that was fine...
Essentially, the thin client can be configured to use one of the 
TransportClient
impl from this package: https://github.com/linkedin/venice/tree/main/clients%2Fvenice-thin-client%2Fsrc%2Fmain%2Fjava%2Fcom%2Flinkedin%2Fvenice%2Fclient%2Fstore%2Ftransport
We should make the FC's transport client likewise configurable
✅ 1
And specifically, use that configured transport for the metadata requests (these are the only ones which are currently hard-coded to require d2, IIUC).
✅ 1
s
Copy code
ClientConfigBuilder<Object, Object, SpecificRecord> clientConfigBuilder =
        new com.linkedin.venice.fastclient.ClientConfig.ClientConfigBuilder<>().setStoreName(storeName)
            .setR2Client(r2Client);

  private AvroGenericStoreClient<String, GenericRecord> getGenericFastClient(
      ClientConfigBuilder<Object, Object, SpecificRecord> clientConfigBuilder,
      MetricsRepository metricsRepository,
      D2Client d2Client) {
    clientConfigBuilder.setStoreMetadataFetchMode(StoreMetadataFetchMode.SERVER_BASED_METADATA);
    clientConfigBuilder.setClusterDiscoveryD2Service(VeniceRouterWrapper.CLUSTER_DISCOVERY_D2_SERVICE_NAME);
    clientConfigBuilder.setMetadataRefreshIntervalInSeconds(1);
    clientConfigBuilder.setD2Client(d2Client);
    clientConfigBuilder.setMetricsRepository(metricsRepository);

    return com.linkedin.venice.fastclient.factory.ClientFactory
        .getAndStartGenericStoreClient(clientConfigBuilder.build());
  }
f
I think the main rub is ensuring the backend has done the correct setup to announce via d2
Once that is done, the client part is not too hard…
s
Oh so here we're not sure backend is configured correctly or not ?
f
I think we’re sure it’s not configured at all 🙃
At least in the docker images it’s not
s
Got it.
f
So either we make it so d2 gets set up automatically by the main code as well… or we make d2 fully optional
Hi, I'm back from vacations today... How are we doing with this issue?
d
Just got back from vacations as well 🙂 Let me know if I can somehow help (once you determine the "fate" of D2 🙂 )
x
Hi everyone sorry for joining this convo 2 months late, was pretty busy recently. I agree with @Felix GV that a d2 only solution in open source might be an overkill. I can help explain the usage of d2 in FC today and we can decide if we want to make some changes to also support the alternative discovery for servers capable of serving metadata requests for FC.
✅ 1
I will try to get a better diagram and explanation when i get a chance to write a wiki about this but here is an attempt for it. I think this d2 discovery can be abstracted out and have different implementations for it that can use other mechanisms such as kubernetes discovery or DNS. Essentially FC just need a discovery service that's capable of taking a store name as input and output a server (ideally a list of servers for highly available HA) that's capable of responding to the metadata fetch requests.
✅ 1
d
Thanks for clarifying the intention. It matches my understanding of its purpose. While it would be great to make it flexible, we don't mind sticking with d2; it's just that we have the matter of configuration to solve. As I mentioned earlier, it appears that the compose does not include the d2 setup, and we'd need it both there and in our "live" deployment 🙂. Are there some reference configurations I can use/customize in these setups?
x
Got it, let me read a little bit about open source d2 (I'm assuming that's what you are using) to make sure there is no gaps and create some instructions and examples on how to set up d2. Just to clarify how does your deployment currently perform store cluster discovery? Is it through d2 already or other means?
f
I think what’s missing is some main code (e.g., in the controller) doing the same crap tests are doing here: https://venicedb.slack.com/archives/C03SLQWRSLF/p1752924885353649?thread_ts=1750259036.062019&amp;channel=C03SLQWRSLF&amp;message_ts=1752924885.353649
I don’t think @Dejan Mijic is using D2 at all yet… he’s just saying they wouldn’t mind using it, if it were easy.
✅ 1
IOW, just like by default we embed the Helix controller into the Venice controller (as opposed to HaaS), we should maybe embed d2 setup out of the box (and at LI we would keep running in the current way, which is that d2 is externally provided).
✅ 1
x
I see, that makes sense.
d
@Felix GV Exactly
🤝 1
x
Having trouble find time to take a stab at this but maybe I can start by sharing some implementation pointers/details on this so people can give feedback and get a better idea on how d2 is intended to be configured for Venice.
✅ 1
d
@Xun Yin Please let me know when you have those ready, thanks 🙏
👍 1
x
Still don't get much time outside of work to focus on this, very sorry for the delays. Here is an issue I started, it basically documents the various configs that are involved and proposing to leverage the D2 set up we do in testing (D2TestUtils) to be part of normal Venice component set up https://github.com/linkedin/venice/issues/2150.
✅ 1
f
Thanks a lot Xun! That’s a great description of everything D2-related. Just one question: is the setup which the test function does naturally idempotent? IOW, would it be fine if the controller systematically setup D2 everytime it started up (e.g. via a new init routine)? Or should it check first that it’s not already setup, and skip if it is?
x
That's a good question. Unfortunately it's not idempotent without additional changes. In fact current implementation will actually explicitly throw exceptions if the d2 cluster/service already exists based on in-memory cache. Regarding to the zk side behavior I need more investigation. Based on initial scan it doesn't look idempotent.
f
If it simply throws and that’s it we’re good… we can catch and swallow
Alternatively, we could make the d2 setup into an admin tool command… then it’s a manual operation to be done once by the operator. Not as great as auto setup, but better than nothing…
x
That might not be a bad idea, personally I think it is best to be bundled together with ZK setup since open source D2 is zk based.
4 Views
Open in Slack
PreviousNext
Powered by