Can you please share here a list of the most important metrics on the read path with the regular client ? Something like: • latency on the server • latency on the router

I am trying to find the best metrics, but I may be using wrong data

Today I have been able to set up some basic Grafana dashboards. These are my numbers so far: • 1 server • 1 router • 1 client, 10 threads, sharing one instance of AvroGenericStoreClient • 1 store • the client accesses only the router using HTTP (no TLS) • same network for all the components • 2.2 ms p50 on the client, 0.6 ms p50 on the router, 0.1ms p50 on the server • 3ms p95 on the client • 22.4 ms p99 on the client, 1.5ms p99 on the router, 0.5ms p99 on the server The difference between the client and the router is huge. I guess that it is due to the AVRO deserialisation. But I would like to understand how can I achieve better numbers. Which are the tunables on the client ? ◦ number of connections ? ◦ turning on HHTP 2.0 ? ◦ do not deserialise ? ◦ should I NOT share the client with multiple threads ? (it looks like we have a great async API, so I assume that having multiple inflight requests is supported, as far as we can open multiple connections or leverage http2 multiplexing)

I have run a new test with 3 routers and 3 servers and replication = 3, but it seems that the results are worse on the client 🤔

The client-side numbers sure look a bit high.

You could try http2, to see if it’s due to connection starvation

how do I enable it ?

@Gaojie Liu, could you help out with this ^

I did more experiments: • the network is not a problem (I see the same results both from a client pod and from the Router pod, also connecting to localhost, instead than passing thru the K8S Service) • Avro is not a problem, there is a “blackhole avro deserializer” option that drops that part I believe that it is more about the number of connections per router. I get better results with only 1 thread in the client

Ok, there is a number of netty workers for the router. IIRC it defaults to twice the core count, but not sure. How many cores does the pod expose to the router process?

on my laptop the default config for the http client is this: selectInterval = 1000 shutdownGracePeriod = 500 interestOpQueued = false ioThreadCount = 10 soTimeout = 0 soReuseAddress = false soLinger = -1 soKeepAlive = false tcpNoDelay = true connectTimeout = 0 sndBufSize = 0 rcvBufSize = 0 backlogSize = 0

Summary for my day: HttpTransportClient is probably too naive in handling connection pooling. So sending concurrent requests doesn’t perform well. TODO: investigate how to tune it

FYI I have tweaked the Venice Client Config to allow to configure the number of connections

ClientConfig clientConfig = ClientConfig.defaultGenericClientConfig(storeName);
clientConfig.setVeniceURL("<http://localhost:7777>");
.....

CloseableHttpAsyncClient httpClient = HttpAsyncClients.custom()
        .setMaxConnPerRoute(10)
        .setMaxConnTotal(10)
        .build();

HttpTransportClient transportClient = new HttpTransportClient(clientConfig.getVeniceURL(),
        httpClient, ClientAuthenticationProvider.DISABLED);

clientConfig.setTransportClient(transportClient);

And now on my laptop I am able to see sub millisecond p50 latencies (0.6ms) and <2ms p99 latencies even with 10 concurrent requests. So I think that there is some tuning to do here

I will be back on this on Monday

Nice

Out of curiosity, what latency targets are datastax looking for?

Frankly speaking, the goal si to be able to say that Venice is generally better than Redis on reads 😉 That said, achieving <1ms p50 and <10ms p99 from a client on the same “network” (cloud AZ) as Venice would be great (maybe @Chris Bartholomew / @Dave Fisher could add some details or refine the goal)

I think competing with Redis is possible but needs further tuning, such as using the PlainTable mode.

I look forward to sneering at that "fastest database" Redis billboard along the 101 here 😂🤣

we Re using HttpClient 4 in the client, and it doesn't support Http 2. https://stackoverflow.com/questions/59527346/is-it-possible-to-make-http-2-requests-with-apache-httpclient-4-2-5 I don't know the requisites in terms of supported jdks. if jdk11 is an option we could switch to using the new HTTP Client bundled with the JDK, that supports HTTP2. Or upgrade the HttpClient dependency to 5

I have a branch with some tweaks on the Venice Client: • upgrade to HttpClient 5 • make the number of connections configurable 1 client, 1 router, TLS encryption (required for HTTP2) With 1 request at a time: 0.21 ms p50 0.42 ms p95 0.63 ms p99 With 10 requests at a time, http2, 1 connection 0.58 1.07

1.38

With 10 requests at a time, http1, 10 connections 0.65 1.13 1.47 With 100 requests at a time, http2, 1 connection 6.80 12.04 14.99 With 100 requests at a time, http1, 10 connections 6.71 9.53 12.99