Hi Team, there’s an error using Presto to query Pinot table. Any idea how to fix it? java.lang.UnsupportedOperationException: Unsupported data table version: 4

Am trying to use the MergeRollupTask on a realtime table to reduce the number of segments. It seems that each task run is failing with an exception like below (not exact copy paste due to limitation). Why are the number of output segment files more than the number of input segments and why do these look to cross partition numbering input was realtime partition 30 output is numbered 0 to 28 if this is the same thing? Thougth the idea of merge rollup was to reduce the number of segments. Also any guidance on the error/how to fix.

Got error status code: 500 (Internal Server Error) with Reason: “Failed to update segment lineage during startReplaceSegments. (TableName = myTable_REALTIME, segmentsFrom = [ myTable__30__2658__20230312T1818Z, myTable__30__2870__20230329T2208Z], segmentsTo = [merged_1day_1681194936985_0_myTable_1382125458330_0, . . . 27 more] )

Hi Team, I am trying to batch ingest parquet files through spark submit.. I can see segment files generated correctly.. However my segments as seen from pinot UI are in BAD state. Can someone please help me here? More details in thread.

Hi team, we added a new column in the schema, and tried to

Reload All Segments

, although the reload status show success, but when we try to query from the table, we could still see the following error

There are xxx invalid segment/s. This usually means that they were created with an older schema. Please reload the table in order to refresh these segments to the new schema.

Please help take a look.

Hi, I am continously getting this error while querying pinot via presto. Can someone please look into this:

failed: CANCELLED: Server sendMessage() failed with Error
io.grpc.StatusRuntimeException: CANCELLED: Server sendMessage() failed with Error
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:648)
	at com.facebook.presto.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:204)
	at com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:295)
	at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:260)
	at com.facebook.presto.operator.Driver.processInternal(Driver.java:426)
	at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:309)
	at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:730)
	at com.facebook.presto.operator.Driver.processFor(Driver.java:302)
	at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1079)
	at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:166)
	at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599)
	at com.facebook.presto.$gen.Presto_0_279_686ef1d____20230408_215611_1.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

@Xiang Fu

Hi team, I have a realtime table and I want to change my kafka cluster to a new one which will not have the previous data(offsets). How can I keep consuming new data coming into this new cluster??

Hi Team, getting segments unavailable on querying a table with replication set to 3 after restarting a server from a pool for 3 and deep store configured, any idea on this?

Hi Team, segment in offline status even after reload is success on server, is controller not able to reconcile the new messages ??

Hi All, I have created a real-time table and have created inverted and range indexes on top of it, I don't see my range index being used in EXPLAIN PLAN . On further analysis, on reloading the segments in Reload Status also I don't see my range index being created. Attaching table definition.

{
  "REALTIME": {
    "tableName": "table_name_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "schemaName": "schema_event",
      "replication": "1",
      "timeColumnName": "audit_timestamp",
      "allowNullTimeValue": true,
      "replicasPerPartition": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant",
      "tagOverrideConfig": {}
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [
        "event_name"
      ],
      "noDictionaryColumns": [],
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.topic.name": "*******",
        "stream.kafka.broker.list": "***********",
        "stream.kafka.consumer.type": "lowlevel",
        "stream.kafka.consumer.prop.auto.offset.reset": "largest",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
        "realtime.segment.flush.threshold.rows": "0",
        "realtime.segment.flush.threshold.time": "6h",
        "realtime.segment.flush.segment.size": "300M",
        "nullHandlingEnabled": "true"
      },
      "rangeIndexColumns": [
        "event_date_skey"
      ],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
      "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": true
    },
    "metadata": {},
    "quota": {},
    "routing": {
      "instanceSelectorType": "strictReplicaGroup"
    },
    "query": {},
    "ingestionConfig": {},
    "isDimTable": false
  }
}

Any way I can merge committed segments for a Realtime table?

Hi Team, Facing issue with sql_lab when querying pinot, it is able to list tables but fails in querying

Can't load plugin: sqlalchemy.dialects:pinot

Have installed pinotdb

Hey guys, good afternoon. We are thinking of doing a migration from 0.6.0 to 0.12.0 using your helm charts but we are facing the following challenge: We have tons of data in a deep store configured in gcs, we wonder how we can make the new installation download and resync the previous tables, as per our knowledge there is no way yet to migrate zookeeper metadata from one cluster into the other and rename the ideal states to make them point to the new component names, for example:

// previous cluster config
"dpt_video_event_captured_v2__0__0__20210212T1650Z": {
  "Server_mls-pinot-server-2.mls-pinot-server-headless.production.svc.cluster.local_8098": "ONLINE",
  "Server_mls-pinot-server-3.mls-pinot-server-headless.production.svc.cluster.local_8098": "ONLINE",
  "Server_mls-pinot-server-4.mls-pinot-server-headless.production.svc.cluster.local_8098": "ONLINE"
}

// new cluster config should be
"dpt_video_event_captured_v2__0__0__20210212T1650Z": {
  "Server_mcls-pinot-server-2.mls-pinot-server-headless.production.svc.cluster.local_8098": "ONLINE",
  "Server_mcls-pinot-server-3.mls-pinot-server-headless.production.svc.cluster.local_8098": "ONLINE",
  "Server_mcls-pinot-server-4.mls-pinot-server-headless.production.svc.cluster.local_8098": "ONLINE"
}

// Note that instead of MLS our new cluster would be named MCLS, this applies also for every resource in the metadata that contains the `mls` prefix, like controllers, servers, brokers, minions, etc.

Do you happen to know of a tool that allows as to do so, or do you have a migration path or guide or anything that could help us for this task? Otherwise wi might need to re ingest the data from Google BigQuery, in the docs I could not find a configuration to ingest data from BigQuery, we could potentially migrate the table from BigQuery using csv files and then ingest it into pinot

Hi team 👋🏼 we have the following field in our schema:

{
  "dataType": "STRING",
  "name": "field",
  "singleValueField": false,
  "maxLength": 32768
},

We expect this field to be a list of large Strings. Can someone confirm in this case is the

maxLength

applied to each String or the total length of the list? Thanks!

Hey guys! I’m having an issue with OFFLINE tables transitioning to an online state when using S3 PinotFS. Realtime tables work perfectly fine, but when we switched from using local to S3 deep storage, Offline table queries start to fail. I’ll add some logs/errors and some more details on what we’re getting to this thread. Thanks!

Hi team, I followed the doc to enable the V2 query engine w/ the POST api and restarted the controllers/brokers/servers, but got below error when querying a table. Does anyone know what went wrong? NVM, upgraded to the 0.12.1 image and it is working now.

[
  {
    "message": "SQLParsingError:\njava.lang.RuntimeException: Error composing query plan for: select * from counterfeit_receipts_test limit 10\n\tat org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:136)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:146)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:127)\n\tat org.apache.pinot.broker.requesthandler.BrokerRequestHandlerDelegate.handleRequest(BrokerRequestHandlerDelegate.java:102)\n...\nCaused by: java.lang.NullPointerException\n\tat org.apache.pinot.query.routing.WorkerManager.assignWorkerToStage(WorkerManager.java:63)\n\tat org.apache.pinot.query.planner.logical.StagePlanner.makePlan(StagePlanner.java:99)\n\tat org.apache.pinot.query.QueryEnvironment.toDispatchablePlan(QueryEnvironment.java:202)\n\tat org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:134)",
    "errorCode": 150
  }
]

Not sure if this is a bug. A quick test on V2 query engine. Most of the time the query succeeded, but sometimes I got below error, and sometimes it returns 0 records (screenshot)

[
  {
    "message": "QueryExecutionError:\njava.lang.RuntimeException: Received error query execution result block: {1000=Stream is already completed, no further calls are allowed\njava.lang.IllegalStateException: Stream is already completed, no further calls are allowed\n\tat org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:444)\n\tat io.grpc.stub.ClientCalls$CallToStreamObserverAdapter.onNext(ClientCalls.java:363)\n\tat org.apache.pinot.query.mailbox.channel.MailboxStatusStreamObserver.send(MailboxStatusStreamObserver.java:53)",
    "errorCode": 200
  }
]

Hi, @Jackie @Mayank I just submitted a pull request #10620 to add read/write lock to RealtimeLuceneTextIndex to keep thread safe, please help us to review it.

hi All, We want to run pinot in production but without using k8. What is the recommended method? is it okay if i run the bin/pinot_admin.sh script for each of the components ?

Hi Team, i am facing some issue with 2 of my dimension tables. After the batch job to refresh the segments, the state of some segments turned ERROR. I checked and the ERROR segments are in server’s local directory. Resetting the segments will turn the segment from ERROR to ONLINE. I was wondering what could cause this issue ? This is happening for only 2/4 dimension tables and non of the OFFLINE/REALTIME tables are impacted. I have attached the server log when the segment became ERROR and the log when i reset the segment. cc: @Xiang Fu @Kartik Khare @Zhengfei

Hi @Jackie, i create a separate thread to avoid useless notifications here. I can't find any message like "_Finish shutting down Pinot server_" in pinot-server logs. In my context, our cluster is running on kubernetes. I've tested a

delete pod

with a longer grace-period (120sec) and i tried a

kill -15

inside the container too: it restarts but no appropriate message. How can i test the expected behavior if a need to restart a pinot-server running in kubernetes without impact ?

Hi, I am looking for a better indexing strategy on columns which have timestamp values stored as long. These columns are used in

Case query

as given below. Basically we are checking if timestamp is null. There are multiple columns in the table like this.

SUM(
  CASE WHEN
    (lastCancelledPassengerTime > 0 AND lastPreallocationCancelTime = 0)
    THEN 1
  ELSE 0 
  END
 )
/  
SUM(
  CASE WHEN lastAllocationTime > 0
  THEN 1
  ELSE 0 
  END
)

PS: can we create the indexing channel separately ?

Hi - I'm digging into an issue with OOM errors and a memory leak of some kind using the java-client and jdbc connections to Pinot. Eventually our Spring Boot application dies with:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncHttpClient-timer-1820-1"

In digging into this I found https://github.com/AsyncHttpClient/async-http-client/issues/1658, the important part (considering I see hundreds of threads marked AsyncHttpClient-timer-X-1) of which is:

AsyncHttpClient-x-x: I/O (twice number of cores)
AsyncHttpClient-timer: timer for timeouts (only one)

Any different number means you're creating multiple clients.

I dug into the Pinot code where this issue seems to be coming from and found where it's creating an AsyncHttpClient: • https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pino[…]/org/apache/pinot/client/JsonAsyncHttpPinotClientTransport.java • https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pino[…]rg/apache/pinot/client/controller/PinotControllerTransport.java • https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java#L120 The problem here as I see it is that the docs for AsyncHttpClient say

AsyncHttpClient instances are intended to be global resources that share the same lifecycle as the application. Typically, AHC will usually underperform if you create a new client for each request, as it will create new threads and connection pools for each.

which the Pinot java client is clearly not doing, and looking at the classes that create these client it's up to other code that's using them to close them. I think this is resulting in client (and their respective threadpools) being kept around in parallel causing memory leakage.

Hi. After upgrading the version to

12.1

,

users

are facing login error (Invalid UserName and Password) on controller, but

Admin

credential works fine. We are using following class in authentication. Brokers are working fine with authentication. Anyone else facing the same issue?

controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.ZkBasicAuthAccessControlFactory

Hi. I'm doing a POC using Pinot and I'm getting this error when trying to ingest data using the ADLS plugin and as deep storage. Can somebody help me to figure out whats happening, pls? Error:

java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
        at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:152) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:129) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]
        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]  
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c7e05a7b58f5435080ab26e9ef8888e2b07dd974]     
Caused by: java.io.FileNotFoundException: /tmp/pinot-bfaa875e-8902-421c-83ff-29eb97361a33/input/56da36da-ba1e-4340-a1c2-362eab3d621e/tags_fake.csv (No such file or directory)
        at java.io.FileOutputStream.open0(Native Method) ~[?:?]
        at java.io.FileOutputStream.open(FileOutputStream.java:298) ~[?:?]
        at java.io.FileOutputStream.<init>(FileOutputStream.java:237) ~[?:?]
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171)
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202)Caused by: java.io.FileNotFoundException: /tmp/pinot-bfaa875e-8902-421c-83ff-29eb97361a33/input/56da36da-ba1e-4340-a1c2-362eab3d621e/tags_fake.csv (No such file or directory)
        at java.base/java.io.FileOutputStream.open0(Native Method)        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
        at org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS.copyToLocalFile(ADLSGen2PinotFS.java:493)
        at org.apache.pinot.spi.filesystem.NoClosePinotFS.copyToLocalFile(NoClosePinotFS.java:98)
        at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.submitSegmentGenTask(SegmentGenerationJobRunner.java:235)
        at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:201)
        at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150)
        ... 13 more

Job-spec yaml:

executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
    segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'
jobType: SegmentCreationAndMetadataPush
inputDirURI: '<adl2://lake/pin_test/>'
outputDirURI: '<adl2://lake/pinot/>'
overwriteOutput: true
pinotFSSpecs:
    - scheme: adl2
      className: org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS
      configs:
        accountName: '<storage_name>'
        accessKey: '<access_key>'
        fileSystemName: '<*>'
recordReaderSpec:
    dataFormat: 'csv'
    className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
    configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
    tableName: 'tags_fake_OFFLINE'
    schemaURI: '<http://localhost:9000/tables/tags_fake_OFFLINE/schema>'
    tableConfigURI: '<http://localhost:9000/tables/tags_fake_OFFLINE>'
pinotClusterSpecs:
    - controllerURI: '<http://localhost:9000>'
includeFileNamePattern: 'glob:**/*.csv'

This message was deleted.

Hi. Does Pinot support environment variables that can be overridden in conf files. e.g. Instead of hardcoding admin user password in

controller.conf

can it be used using environment variable. If yes, then what is the name of environment variable ? Is there any list somewhere in documentation ? Thanks

After redeploying the k8s nodes that the pinot cluster runs on, we are seeing the following error:

[
  {
    "message": "org.apache.pinot.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinot-server-3.pinot-server-headless.pinot.svc.cluster.local/x.x.x.x:8098\nCaused by: java.net.ConnectException: Connection refused\n\tat java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)\n\tat org.apache.pinot.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)\n\tat org.apache.pinot.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)",
    "errorCode": 425
  },
  {
    "message": "6 servers [pinot-server-0_R, pinot-server-3_R, pinot-server-2_R, pinot-server-5_R, pinot-server-4_R, pinot-server-1_R] not responded",
    "errorCode": 427
  }
]

Hi, I am trying to debug slow queries & query exceptions. • Most of the time query is waiting in queue to be executed on server node • Scheduling method used is default

FCFS

. • May I know which method is recommended? and if that will solve the problem • Logs and cluster stats in comments

Hi All, while building table using TableBuilder class

private String _segmentPushType = DEFAULT_SEGMENT_PUSH_TYPE;

as this deprecated, can you make sure it is not set…by default it is getting set. Can you help me in how to not add this config?

hey guys @channel, I tried to implement deep storage on s3 in pinot using the official doc but facing difficulties. below are my setup details followed by the issue I am facing. Setup: used below mentioned helm chart to deploy pinot on my eks cluster:

utkarsh2811@ip-192-168-0-112 ~ % helm list -n pinot-ds
NAME 	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART      	APP VERSION
pinot	pinot-ds 	1       	2023-04-20 12:58:45.951593 +0530 IST	deployed	pinot-0.2.6	0.2.6      
utkarsh2811@ip-192-168-0-112 ~ %

applied below value files with custom settings for s3 deep storage for creating pinot cluster:

utkarsh2811@ip-192-168-0-112 ~ % cat work/helm/pinot/pinot-custom-values.yaml 
controller:
  data:
    dir: <s3://data-pinot-deep-storage>

  extra:
    configs: |-
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true
      controller.local.temp.dir=/tmp/pinot-tmp-data/
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.controller.storage.factory.s3.region=ap-south-1
      pinot.controller.segment.fetcher.protocols=file,http,s3
      pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

server:
  extra:
    configs: |-
      pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.server.storage.factory.s3.region=ap-south-1
      pinot.server.segment.fetcher.protocols=file,http,s3
      pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
utkarsh2811@ip-192-168-0-112 ~ %

Issue: whenever i tried to ingest a data topic in realtime tables from kafka, they tend to go on forever and never really complete. neither i see any segment files being created at my s3 bucket. here are the few logs from controller: controller logs:

Caught exception while uploading segment: pinot_data_view__0__0__20230420T1035Z from instance: Server_10.1.171.4_8098
software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 8Z5HM8K2PRH4P5H2, Extended Request ID: PqTaeFo+KieHk0/3b8VVtNDrT9NQO2Q1Q5y1Ks9FjC3HUMasZHUUviHMzvD+pI2RQBgcmsgrMYc=)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:8123) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at software.amazon.awssdk.services.s3.S3Client.putObject(S3Client.java:15265) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at org.apache.pinot.plugin.filesystem.S3PinotFS.copyFromLocalFile(S3PinotFS.java:581) ~[pinot-s3-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at org.apache.pinot.spi.filesystem.NoClosePinotFS.copyFromLocalFile(NoClosePinotFS.java:110) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
	at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentUpload(LLCSegmentCompletionHandlers.java:372) ~

any idea whats wrong here?