Can you please share here a list of the most important metrics on the read path with the regular client ? Something like: • latency on the server • latency on the router

I am trying to find the best metrics, but I may be using wrong data

Today I have been able to set up some basic Grafana dashboards. These are my numbers so far: • 1 server • 1 router • 1 client, 10 threads, sharing one instance of AvroGenericStoreClient • 1 store • the client accesses only the router using HTTP (no TLS) • same network for all the components • 2.2 ms p50 on the client, 0.6 ms p50 on the router, 0.1ms p50 on the server • 3ms p95 on the client • 22.4 ms p99 on the client, 1.5ms p99 on the router, 0.5ms p99 on the server The difference between the client and the router is huge. I guess that it is due to the AVRO deserialisation. But I would like to understand how can I achieve better numbers. Which are the tunables on the client ? ◦ number of connections ? ◦ turning on HHTP 2.0 ? ◦ do not deserialise ? ◦ should I NOT share the client with multiple threads ? (it looks like we have a great async API, so I assume that having multiple inflight requests is supported, as far as we can open multiple connections or leverage http2 multiplexing)

I have run a new test with 3 routers and 3 servers and replication = 3, but it seems that the results are worse on the client 🤔

The client-side numbers sure look a bit high.

You could try http2, to see if it’s due to connection starvation

how do I enable it ?

@Gaojie Liu, could you help out with this ^

I did more experiments: • the network is not a problem (I see the same results both from a client pod and from the Router pod, also connecting to localhost, instead than passing thru the K8S Service) • Avro is not a problem, there is a “blackhole avro deserializer” option that drops that part I believe that it is more about the number of connections per router. I get better results with only 1 thread in the client

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

I will be back on this on Monday

Nice

Out of curiosity, what latency targets are datastax looking for?

Frankly speaking, the goal si to be able to say that Venice is generally better than Redis on reads 😉 That said, achieving <1ms p50 and <10ms p99 from a client on the same “network” (cloud AZ) as Venice would be great (maybe @Chris Bartholomew / @Dave Fisher could add some details or refine the goal)

I think competing with Redis is possible but needs further tuning, such as using the PlainTable mode.

I look forward to sneering at that "fastest database" Redis billboard along the 101 here 😂🤣

This message was deleted.

This message was deleted.

This message was deleted.