has anyone seen the client become the bottleneck for their queries before? i’m seeing

timeMs=~1000ms

on my broker, but it takes 2/3 seconds on python/java clients on that single line to execute the query e.g.

# python
curs.execute(sql_query)

# java
ResultSetGroup pinotResultSetGroup = pinotConnection.execute(pinotClientRequest);

hey friends, we are trying to setup pre aggregation for one of realtime tables in our cluster for this schema

{
  "schemaName": "etsyads_metrics",
  "dimensionFieldSpecs": [
    {
      "name": "shop_id",
      "dataType": "LONG"
    },
    {
      "name": "listing_id",
      "dataType": "LONG"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "cost",
      "dataType": "INT",
      "defaultNullValue": 0
    },
    {
      "name": "impression_count",
      "dataType": "INT",
      "defaultNullValue": 0
    },
    {
      "name": "click_count",
      "dataType": "INT",
      "defaultNullValue": 0
    },
    {
      "name": "order_count",
      "dataType": "INT",
      "defaultNullValue": 0
    },
    {
      "name": "revenue",
      "dataType": "INT",
      "defaultNullValue": 0
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "serve_time",
      "dataType": "LONG",
      "format": "1:SECONDS:EPOCH",
      "granularity": "1:HOURS"
    }
  ]
}

we were reading this https://docs.pinot.apache.org/basics/components/table#pre-aggregation and we set this as our noDictionaryMetrics

"noDictionaryColumns": [
      "click_count",
      "order_count",
      "impression_count",
      "cost",
      "revenue"
    ],
    "aggregateMetrics" : true

but we still see multiple rows for the same dimension values in the same hour, does anyone have some guidance as to why this may be happening?

Hey, I'm having an issue with hybrid tables. It looks like RealtimeToOfflineSegmentsTask was not moving all the records in the realtime segment, son for example I've got 4mill records in the realtime table but I end up with only 600 in the offline table after the realtime segments are deleted. Surely this is a configuration issue in my table. I was wondering whether this could be a problem between the table retention periods and the offline flow buffer configuration.

Hey friends, I am trying to convert timezone based on location field with transformation functions, but the real-time table does not accept the config and after save the page goes blank..

"ingestionConfig": {

"filterConfig": null,

"transformConfigs": [

{

"columnName": "e_time_stamp",

"transformFunction": "Groovy({timestamp + ' Australia/Melbourne'}, timestamp)"

},

{

"columnName": "epoc_time_stamp",

"transformFunction": "fromDateTime(e_time_stamp, 'yyyy-MM-dd''T''HH:mm:ssZZZ')"

},

{

"columnName": "a_time_stamp",

"transformFunction": "Groovy({state == 'QLD' ? DATETIMECONVERT(epoc_time_stamp, '1:MILLISECONDS:EPOCH', '1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd''T''HH:mm:ss tz(Australia/Queensland)', '1:SECONDS') : DATETIMECONVERT(epoc_time_stamp, '1:MILLISECONDS:EPOCH', '1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd''T''HH:mm:ss tz(Australia/Melbourne)', '1:SECONDS')}, state)"

}

]

},

Hi Folks, We have a field of LONG data type which was earlier mentioned as "dimensionField" in our schema. This field stored the time in epoch milliseconds. Using this schema the data was loaded. However, we now want to make this field "metricField" so that we can apply range indexing to it. When we tried to update the schema it resulted in the error

Backward incompatible schema <schemaName>.Only allow adding new columns

What can be the way to proceed ?

Hi Friends, has anyone had issue with pinot connector with trino: I am getting the following Errors and the query never runs:

Caught exception while deserializing data table of size: 1505 from server: 10.12.2.133_R

java.lang.UnsupportedOperationException: Unsupported data table version: 3

2021-09-13T06:35:32.016Z        INFO    dispatcher-query-15     io.trino.event.QueryMonitor     TIMELINE: Query 20210913_063531_00006_7vsrs :: Transaction:[71205803-47f0-475d-88d4-56d5b4719d14] :: elapsed 220ms :: planning 46ms :: waiting 5ms :: scheduling 31ms :: running 20ms :: finishing 123ms :: begin 2021-09-13T06:35:31.783Z :: end 2021-09-13T06:35:32.003Z

2021-09-13T06:35:37.244Z        INFO    Query-20210913_063537_00007_7vsrs-464   io.trino.plugin.pinot.PinotSplitManager Got routing table for speedtest: {speedtest_REALTIME={Server_10.12.2.133_8098=[speedtest__0__0__20210913T0339Z]}}

2021-09-13T06:35:37.256Z        INFO    20210913_063537_00007_7vsrs.1.0-0-135   io.trino.plugin.pinot.PinotSegmentPageSource    Query 'SELECT upload_throughput, e_time_stamp, cpeid, download_throughput, avcid, service_plan, state, epoc_time_stamp FROM speedtest_REALTIME  LIMIT 10' on host 'Optional[Server_10.12.2.133_8098]' for segment splits: [speedtest__0__0__20210913T0339Z]

2021-09-13T06:35:37.273Z        ERROR   nioEventLoopGroup-2-2   org.apache.pinot.core.transport.DataTableHandler        Caught exception while deserializing data table of size: 1505 from server: 10.12.2.133_R

java.lang.UnsupportedOperationException: Unsupported data table version: 3

at org.apache.pinot.core.common.datatable.DataTableFactory.getDataTable(DataTableFactory.java:37)

at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:67)

at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:36)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.base/java.lang.Thread.run(Thread.java:829)

2021-09-13T06:36:37.263Z        ERROR   SplitRunner-28-135      io.trino.execution.executor.TaskExecutor        Error processing Split 20210913_063537_00007_7vsrs.1.0-0 PinotSplit{splitType=SEGMENT, segments=[speedtest__0__0__20210913T0339Z], segmentHost=Optional[Server_10.12.2.133_8098]} (start = 4.476924908050052E9, wall = 60007 ms, cpu = 0 ms, wait = 0 ms, calls = 1)

java.lang.NullPointerException: null value in entry: Server_10.12.2.133_8098=null

at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:32)

at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:42)

at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:72)

at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:119)

at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:454)

at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:433)

at io.trino.plugin.pinot.PinotSegmentPageSource.queryPinot(PinotSegmentPageSource.java:221)

at io.trino.plugin.pinot.PinotSegmentPageSource.fetchPinotData(PinotSegmentPageSource.java:182)

at io.trino.plugin.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:150)

at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)

at io.trino.operator.Driver.processInternal(Driver.java:387)

at io.trino.operator.Driver.lambda$processFor$9(Driver.java:291)

at io.trino.operator.Driver.tryWithLock(Driver.java:683)

at io.trino.operator.Driver.processFor(Driver.java:284)

at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)

at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)

at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)

at io.trino.$gen.Trino_361____20210913_051507_2.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

2021-09-13T06:36:37.265Z        ERROR   remote-task-callback-13 io.trino.execution.StageStateMachine    Stage 20210913_063537_00007_7vsrs.1 failed

java.lang.NullPointerException: null value in entry: Server_10.12.2.133_8098=null

at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:32)

at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:42)

at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:72)

at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:119)

at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:454)

at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:433)

at io.trino.plugin.pinot.PinotSegmentPageSource.queryPinot(PinotSegmentPageSource.java:221)

at io.trino.plugin.pinot.PinotSegmentPageSource.fetchPinotData(PinotSegmentPageSource.java:182)

at io.trino.plugin.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:150)

at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)

at io.trino.operator.Driver.processInternal(Driver.java:387)

at io.trino.operator.Driver.lambda$processFor$9(Driver.java:291)

at io.trino.operator.Driver.tryWithLock(Driver.java:683)

at io.trino.operator.Driver.processFor(Driver.java:284)

at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)

at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)

at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)

at io.trino.$gen.Trino_361____20210913_051507_2.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

2021-09-13T06:36:37.280Z        INFO    dispatcher-query-15     io.trino.event.QueryMonitor     TIMELINE: Query 20210913_063537_00007_7vsrs :: Transaction:[e51637ae-dc8c-409b-a9be-fe544757a91a] :: elapsed 60076ms :: planning 55ms :: waiting 3ms :: scheduling 12ms :: running 60007ms :: finishing 2ms :: begin 2021-09-13T06:35:37.189Z :: end 2021-09-13T06:36:37.265Z

================================================================

trino:default> SHOW TABLES FROM pinot.default;

Table

------------

speedtest

speedtest2

speedtest3

(3 rows)

Query 20210913_063531_00006_7vsrs, FINISHED, 1 node

Splits: 19 total, 19 done (100.00%)

0.22 [4 rows, 103B] [18 rows/s, 468B/s]

trino:default> select * from speedtest limit 10;

Query 20210913_063537_00007_7vsrs, FAILED, 1 node

Splits: 18 total, 0 done (0.00%)

1:00 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20210913_063537_00007_7vsrs failed: null value in entry: Server_10.12.2.133_8098=null

Hi, it seems that

bin/pinot-ingestion-job.sh

does not accept yaml ingestion job config. It does not work with the example command (

bin/pinot-ingestion-job.sh examples/batch/airlineStats/ingestionJobSpec.yaml

) I copy from official doc. It will shows following error

Error: No argument is allowed: examples/batch/airlineStats/ingestionJobSpec.yaml

. Can someone help me what's wrong here ?

do we need to increase jvm xmx limit ?

Hey all! We have a metric column that we’re unsure is indexed correctly. Running

SELECT metric_column FROM table ORDER BY metric_column DESC LIMIT 10

results in a full table scan. So far we’ve followed the instructions for setting up a raw value forward index, but it doesn’t seem to help.

Hello, I'm trying to configure presto to query pinot tables. The catalogue seems fine, I can show tables in Pinot. However when I issue a query I get the following error:

Query <id> failed: Cannot fetch from cache <table>

Any hints to fix this error would be appreciated

Is there any way to check slow queries in pinot ?

Can anyone help me to understand, what this error means -

Error: Could not find or load main class org.apache.pinot.thirdeye.anomaly.ThirdEyeAnomalyApplication

. And how to fix this, don't have much experience in Java.

Hi, I have a question about

DATETIMECONVERT

. It mentions it buckets the time based on the given time granularity, but what is the start of the first bucket? e.g. when running this query:

SELECT DATETIMECONVERT(time_col, '1:MILLISECONDS:EPOCH', '1:SECONDS:EPOCH', '30:DAYS') as new_time_col, COUNT(id) FROM table WHERE (time_col BETWEEN <epoch_second of 7/16> AND <epoch_second of 9/16>) AND GROUP BY new_time_col ORDER BY new_time_col

It returns 3 buckets for: 7/1, 7/31 and 8/30. So I wonder how is this being calculated?

Hello, I have an OFFLINE table with a configured retention:

"segmentsConfig": {
      "replication": "1",
      "timeColumnName": "timestamp",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "90",
      "timeType": "MILLISECONDS",
      "schemaName": "schema_607ec74cd839000300105f34"
    }

But my segments aren’t deleted, like this one:

{
  "segment.offline.download.url": "<s3://pinot-production-store/datasource_607ec74cd839000300105f34/datasource_607ec74cd839000300105f34_1618921448331_1618947468383_1>",
  "segment.start.time": "1618921448331",
  "segment.time.unit": "MILLISECONDS",
  "segment.end.time": "1618947468383",
  "segment.total.docs": "390000",
  "segment.table.name": "datasource_607ec74cd839000300105f34",
  "segment.creation.time": "1619049989622",
  "segment.name": "datasource_607ec74cd839000300105f34_1618921448331_1618947468383_1",
  "segment.index.version": "v3",
  "segment.offline.refresh.time": "-9223372036854775808",
  "segment.type": "OFFLINE",
  "segment.offline.push.time": "1619050008800",
  "segment.crc": "1089143347"
}

I see in the logs the following message:

2021/09/17 03:45:12.752 INFO [RetentionManager] [pool-9-thread-6] Start managing retention for table: datasource_607ec74cd839000300105f34_OFFLINE

And except this log, no other log message mention the RetentionManager Do you know why or how I can debug this? Thank you

Hi Pinot experts, we had a sudden ingestion lag (> 45m) in one of our tables today without any increase in incoming throughput. This happened suddenly to just that one table (the rest of the tables had very little lag in seconds). We had to scale up our Kafka partitions for it to catch up. What might be the possible reason behind this sudden lag? @Tanmay Movva

does anyone have an example to make the recommendation engine works with realtime tables for segmentSizeRecommendations? I keep on trying to running it but I keep on getting:

"segmentSizeRecommendations": {
    "numRowsPerSegment": 0,
    "numSegments": 0,
    "segmentSize": 0,
    "message": "Segment sizing for realtime-only tables is done via Realtime Provisioning Rule"
  },

Hi, does anyone know why the

/version

REST API endpoint would return an empty object? The other endpoints work for me.

Hi All Can superset support auth for pinot, I don’t find some auth configurations in superset? thx

Hi pinot experts, I tried to upload a segment with the following command:

pinot-admin.sh LaunchDataIngestionJob -jobSpecFile batch_ingestion.yaml

and I can see the output shows it returned 200 and thought it was successful:

Start pushing segments: 
...
Response for pushing table enriched_invoices_experiment segment enriched_invoices_experiment_OFFLINE_1595614484_1631217493_0 to location <http://10.124.9.8:9000> - 200: {"status":"Successfully uploaded segment: enriched_invoices_experiment_OFFLINE_1595614484_1631217493_0 of table: enriched_invoices_experiment"}

However, I don’t see it showing up by querying the table. Could you please give some pointers where I could look up the issue? I checked the logs and didn’t find errors.

Hi everyone , I am trying to run the spark ingestion job with apache pinot 0.8.0 in my own cluster setup. I am able to run the standalone job, but when am trying to run spark ingestion job it is giving me following error: java.lang.RuntimeException: Failed to create IngestionJobRunner instance for class - org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner I am using the below command to run the spark job: ${SPARK_HOME}/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master "local[2]" --deploy-mode client --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar" local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar -jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/transcript/transcript_local_jobspec.yaml I am also attaching screenshot of the error and my job spec file for better understanding. Could anyone please help with the same ?

Hello Freinds, can you help me please, If I turn off the PC everything I did is gone. So If I connect to my machine how can I run ThirdEye? I already install it and go to the first page but now i want to re-run it but i don't know how.

if you use star tree index and say you have a time column, if you wanted to do different aggregations based on time, does that mean that the time column also has to be part of the indexes for the star tree? say user_id , click_count, serve_time then I should do user_id, serve_time and then SUM(click_count) as part as the aggregation

Hello team, I’m trying to query JSON data. I have a field called

category

with a JsonArray, for example:

["category1", "category2", "category3"]

and I need to get all rows where

category

contains

category2

I was looking at https://docs.pinot.apache.org/users/user-guide-query/json-queries but I don’t see any way for searching inside a JsonArray. Any idea how it could be done?

I added a new table yesterday to my pinot cluster I wonder what could have caused this metrics to go up a lot

Controllers don’t need it if you use URI based segment push

Ephemeral storage is fine as long as you plan for startup of a new server - delay while segments get pulled from deep store

Hello, i wonder if there is any guidance on troubleshoot ingestion issues. We have a table set up consuming from Kafka, but nothing is ingested. And there is no such info/warn/error events showing up in the logs either.

Hello guys, any ideas why pinot isnt reading all messages from my kafka topic? According to the logs, it has reached the 3815366 offset , which is the amount of messages I have. However, when I query them on pinot, i got only 3794657

pinot-server_1  | Consumed 0 events from (rate:0.0/s), currentOffset=3815366, numRowsConsumedSoFar=590061, numRowsIndexedSoFar=577470

Hi guys, we faced a sudden ingestion rate drop to 0 some time back. These are our observations (have attached the log files). Realtime-server logs:

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
 15         at org.apache.pinot.core.realtime.impl.dictionary.BaseOffHeapMutableDictionary.indexValue(BaseOffHeapMutableDictionary.java:361)
 14         at org.apache.pinot.core.realtime.impl.dictionary.StringOffHeapMutableDictionary.index(StringOffHeapMutableDictionary.java:61)
 13         at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:521)
 12         at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
 11         at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
 10         at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
  9         at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
  8         at java.lang.Thread.run(Thread.java:748)
  7 2021/09/23 15:00:16.067 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceSta    tusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=163240921557    7, version=11, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
  6 2021/09/23 15:00:26.077 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceSta    tusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=16324091    07080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init    ;;
  5 2021/09/23 15:00:36.093 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceSta    tusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=16324091    07080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init    ;;
  4 2021/09/23 15:00:46.103 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceSta    tusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=16324091    07080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init    ;;
  3 2021/09/23 15:00:56.113 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceSta    tusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=16324092    55125, version=22, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init    ;;
  2 2021/09/23 15:01:06.122 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceSta    tusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23040__20210923T0619Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=16324092    66062, version=48, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init    ;;
  1 2021/09/23 15:01:16.215 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Service status is GOOD after 190391ms
  0 2021/09/23 15:01:16.222 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Pinot server ready
  1 2021/09/23 15:30:06.057 WARN [ZkBaseDataAccessor] [ZkClient-EventThread-23-apache-pinot-zookeeper-bitnami-headless.pinot.svc.cluster.local:2181] Fail to read record for paths: {/pinot-prod/INSTANCES/S    erver_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/74dc1531-5db0-47a4-b97c-95cabc4067dc=-101}

I am trying to get Pinot running on Kubernetes. After installing with Helm, there is this note. Where is the

StorageClass

specified? This is on bare metal, btw.

Please specify StorageClass based on your cloud vendor.