Was doing a PT on the pinot setup 3 brokers 3 servers 3 cont Apache Pinot #general

Was doing a PT on the pinot setup ( 3 brokers, 3 s...

Shounak Kulkarni

04/14/2020, 6:25 PM

Was doing a PT on the pinot setup ( 3 brokers, 3 servers, 3 controllers). Currently there are 20,000,000+ entries (64 segments over all 3 servers 35MB as segment size ). The topic in kafka has 3 partitions and replicationPerPartition is 2 in pinot. With a minimal load of 100 rps the brokers are going down. logs given below

Shounak Kulkarni

04/15/2020, 11:06 AM

The pods are stable now but latency is very poor. For 80rps latency was 5ms but with 100rps its going beyond 1s. No error logs on any container.

Kishore G

04/15/2020, 1:47 PM

What’s the kubernetes container setting. Have you added any indexes?

Shounak Kulkarni

04/15/2020, 1:48 PM

Yes, I have applied inverted index to the columns used in where clause of queries

Shounak Kulkarni

04/15/2020, 1:53 PM

broker - 4 GB, 1200 millicores server - 5 GB, 1500 millicores controller- 3 GB, 500 millicores

Kishore G

04/15/2020, 1:58 PM

Can you paste a sample query?

Kishore G

04/15/2020, 1:59 PM

And response

Kishore G

04/15/2020, 2:05 PM

Also, you are probably hitting capacity limit. 1.5 core is not enough to do performance testing.

Shounak Kulkarni

04/15/2020, 2:10 PM

select MAX(timeSinceEpoch), modelYearCode FROM vehicleTable where timeSinceEpoch >= 1584988200 and userId = 880560022 group by modelYearCode limit 10

Copy code

{
  "resultTable": {
    "dataSchema": {
      "columnDataTypes": [
        "DOUBLE",
        "STRING"
      ],
      "columnNames": [
        "max(timeSinceEpoch)",
        "modelYearCode"
      ]
    },
    "rows": [
      [
        1585835137,
        "CUJ202008"
      ],
      [
        1585717081,
        "CUJ202009"
      ],
      [
        1585741681,
        "CUJ202010"
      ],
      [
        1585800122,
        "CUJ202011"
      ],
      [
        1585839286,
        "CUJ202003"
      ],
      [
        1585828710,
        "CUJ202014"
      ]
    ]
  },
  "exceptions": [],
  "numServersQueried": 3,
  "numServersResponded": 3,
  "numSegmentsQueried": 38,
  "numSegmentsProcessed": 15,
  "numSegmentsMatched": 15,
  "numConsumingSegmentsQueried": 3,
  "numDocsScanned": 309,
  "numEntriesScannedInFilter": 349,
  "numEntriesScannedPostFilter": 618,
  "numGroupsLimitReached": false,
  "totalDocs": 22051919,
  "timeUsedMs": 5,
  "segmentStatistics": [],
  "traceInfo": {},
  "minConsumingFreshnessTimeMs": 9223372036854776000
}

Shounak Kulkarni

04/15/2020, 2:13 PM

regarding hitting capacity the cpu usage for pods is not going even beyond 1 core. Resource utilization during PT :

Copy code

NAME                 CPU(cores)   MEMORY(bytes)
pinot-broker-0       179m         994Mi
pinot-broker-1       36m          1006Mi
pinot-broker-2       28m          1004Mi
pinot-controller-0   4m           913Mi
pinot-controller-1   4m           926Mi
pinot-controller-2   10m          944Mi
pinot-server-0       70m          1481Mi
pinot-server-1       124m         1477Mi
pinot-server-2       127m         1475Mi
pinot-zookeeper-0    14m          382Mi

Kishore G

04/15/2020, 2:13 PM

do you have inverted index or sorted index on userId

Shounak Kulkarni

04/15/2020, 2:14 PM

inverted index

Kishore G

04/15/2020, 2:15 PM

try sorted index

Kishore G

04/15/2020, 2:16 PM

but even this query is using 5ms

Kishore G

04/15/2020, 2:16 PM

so even if you run sequentially, you should be getting 100+ qps

Shounak Kulkarni

04/15/2020, 2:17 PM

so that "timeUsedMs" field specifies exactly which time?

Kishore G

04/15/2020, 2:17 PM

total time at the broker before responding

Kishore G

04/15/2020, 2:18 PM

https://docs.pinot.apache.org/pinot-user-guide/response-format

👍 1

Shounak Kulkarni

04/15/2020, 2:19 PM

so the latency showed by the PT tool (270ms) is all network latency?

Kishore G

04/15/2020, 2:20 PM

yes, where are you running the PT tool

Shounak Kulkarni

04/15/2020, 2:21 PM

on a separate vm

Kishore G

04/15/2020, 2:21 PM

yeah, your client might not be scaling

👍 1

Kishore G

04/15/2020, 2:22 PM

or network latency

Shounak Kulkarni

04/15/2020, 2:30 PM

Is it possible that traffic congestion is happening at brokers?

Kishore G

04/15/2020, 2:31 PM

not at this qps

Kishore G

04/15/2020, 2:32 PM

I dont know if kube is limiting the network io

Shounak Kulkarni

04/15/2020, 2:32 PM

yes even resources are not getting used by broker that much

Shounak Kulkarni

04/15/2020, 2:34 PM

I'll check the kube side

Kishore G

04/15/2020, 2:35 PM

is this real-time?

Shounak Kulkarni

04/15/2020, 2:36 PM

Yes table type is realtime but currently working with static generated entries in kafka

Kishore G

04/15/2020, 2:39 PM

thats ok

Open in Slack

Previous Next