i used

pinot-admin.sh AddTable

, specified table and schema json

update: if i ingest from query console like this, this issue is gone:

SET taskName='mytask';
SET input.fs.className='org.apache.pinot.plugin.filesystem.S3PinotFS';
SET includeFileNamePattern='glob:**/*.jsonl';
SET inputFormat='json';
INSERT INTO transcript
FROM FILE 's3://<bucket>/pinot-transcript-input/'

this issue occurs if i create a batch ingestion job using

pinot-admin.sh LaunchDataIngestionJob -jobSpecFile=./job-spec.yml

is anything problematic in job spec?

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'

jobType: SegmentCreationAndMetadataPush
inputDirURI: 's3://<bucket>/pinot-transcript-input/'
includeFileNamePattern: 'glob:**/*.jsonl'
outputDirURI: 's3://<bucket>/pinot-transcript-output/'
overwriteOutput: true
pinotFSSpecs:
  - scheme: s3
    className: org.apache.pinot.plugin.filesystem.S3PinotFS
    configs:
      region: 'us-west-2'
recordReaderSpec:
  dataFormat: 'json'
  className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader'
tableSpec:
  tableName: 'transcript'
  schemaURI: 'https://<CONTROLLER>/tables/transcript/schema'
  tableConfigURI: 'https://<CONTROLLER>/tables/transcript'
pinotClusterSpecs:
  - controllerURI: 'https://<CONTROLLER>'
pushJobSpec:
  pushAttempts: 2
  pushRetryIntervalMillis: 1000
  pushParallelism: 2

image.png

hmm found something interesting in server log:

Trying to enable the forward index
 for a column subject missing either the dictionary (disabled) and / or 
the inverted index (disabled) is not possible. Either a refresh or 
back-fill is required to get the forward index, ignoring

insert into

is executed from the minion, which should use same code path of local dist if the version is the same

Server log:

java.lang.IllegalStateException: Forward index disabled for column: studentID, cannot create DataFetcher!

	at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:601) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.common.DataFetcher.<init>(DataFetcher.java:72) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.operator.ProjectionOperator.<init>(ProjectionOperator.java:49) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.operator.ProjectionOperatorUtils$DefaultImplementation.getProjectionOperator(ProjectionOperatorUtils.java:54) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.operator.ProjectionOperatorUtils.getProjectionOperator(ProjectionOperatorUtils.java:39) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.plan.ProjectPlanNode.run(ProjectPlanNode.java:81) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.plan.SelectionPlanNode.run(SelectionPlanNode.java:71) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.plan.CombinePlanNode.getCombineOperator(CombinePlanNode.java:92) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:79) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:44) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:51) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.executeInternal(ServerQueryExecutorV1Impl.java:389) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.executeInternal(ServerQueryExecutorV1Impl.java:259) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.execute(ServerQueryExecutorV1Impl.java:136) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.query.executor.QueryExecutor.execute(QueryExecutor.java:59) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:142) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:124) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]

	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]

	at org.apache.pinot.shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at org.apache.pinot.shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]

	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]

also server log:

2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index
 for a column fdcSummaries missing either the dictionary (disabled) and /
 or the inverted index (disabled) is not possible. Either a refresh or 
back-fill is required to get the forward index, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index
 for a column firstName missing either the dictionary (disabled) and / 
or the inverted index (disabled) is not possible. Either a refresh or 
back-fill is required to get the forward index, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index for a sorted column gender, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index
 for a column lastName missing either the dictionary (disabled) and / or
 the inverted index (disabled) is not possible. Either a refresh or 
back-fill is required to get the forward index, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index
 for a column score missing either the dictionary (disabled) and / or 
the inverted index (disabled) is not possible. Either a refresh or 
back-fill is required to get the forward index, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index for a sorted column studentID, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index
 for a column subject missing either the dictionary (disabled) and / or 
the inverted index (disabled) is not possible. Either a refresh or 
back-fill is required to get the forward index, ignoring
2024/05/10 19:08:45.917 WARN [ForwardIndexHandler] [HelixTaskExecutor-message_handle_thread_0] Trying to enable the forward index
 for a column timestampInEpoch missing either the dictionary (disabled) 
and / or the inverted index (disabled) is not possible. Either a refresh
 or back-fill is required to get the forward index, ignoring

same issue when i run from within docker 😢

can you untar gz the segment file and check the indexmap and metadata.properties file

yea forward index is there.....

metadata:

segment.padding.character = \u0000
segment.name = transcript_OFFLINE_0
segment.table.name = transcript
segment.dimension.column.names = fdcSummaries,firstName,gender,lastName,studentID,subject
segment.metric.column.names = score
segment.datetime.column.names = timestampInEpoch
segment.total.docs = 5
custom.input.data.file.uri = s3://<bucket>/pinot-transcript-input/transcript.json
column.fdcSummaries.cardinality = 5
column.fdcSummaries.totalDocs = 5
column.fdcSummaries.dataType = JSON
column.fdcSummaries.bitsPerElement = 3
column.fdcSummaries.lengthOfEachEntry = 31
column.fdcSummaries.columnType = DIMENSION
column.fdcSummaries.isSorted = false
column.fdcSummaries.hasDictionary = true
column.fdcSummaries.isSingleValues = true
column.fdcSummaries.maxNumberOfMultiValues = 0
column.fdcSummaries.totalNumberOfEntries = 5
column.fdcSummaries.isAutoGenerated = false
column.fdcSummaries.minValue = {"fdc2":42\u0000\u0000"fdc1":"foo"}
column.fdcSummaries.maxValue = {"fdc4":"world"\u0000\u0000"fdc3":"hello"}
column.fdcSummaries.defaultNullValue = null
column.firstName.cardinality = 4
column.firstName.totalDocs = 5
column.firstName.dataType = STRING
column.firstName.bitsPerElement = 2
column.firstName.lengthOfEachEntry = 4
column.firstName.columnType = DIMENSION
column.firstName.isSorted = false
column.firstName.hasDictionary = true
column.firstName.isSingleValues = true
column.firstName.maxNumberOfMultiValues = 0
column.firstName.totalNumberOfEntries = 5
column.firstName.isAutoGenerated = false
column.firstName.minValue = Bob
column.firstName.maxValue = Nick
column.firstName.defaultNullValue = null
column.gender.cardinality = 2
column.gender.totalDocs = 5
column.gender.dataType = STRING
column.gender.bitsPerElement = 1
column.gender.lengthOfEachEntry = 6
column.gender.columnType = DIMENSION
column.gender.isSorted = true
column.gender.hasDictionary = true
column.gender.isSingleValues = true
column.gender.maxNumberOfMultiValues = 0
column.gender.totalNumberOfEntries = 5
column.gender.isAutoGenerated = false
column.gender.minValue = Female
column.gender.maxValue = Male
column.gender.defaultNullValue = null
column.lastName.cardinality = 4
column.lastName.totalDocs = 5
column.lastName.dataType = STRING
column.lastName.bitsPerElement = 2
column.lastName.lengthOfEachEntry = 5
column.lastName.columnType = DIMENSION
column.lastName.isSorted = false
column.lastName.hasDictionary = true
column.lastName.isSingleValues = true
column.lastName.maxNumberOfMultiValues = 0
column.lastName.totalNumberOfEntries = 5
column.lastName.isAutoGenerated = false
column.lastName.minValue = Doe
column.lastName.maxValue = Young
column.lastName.defaultNullValue = null
column.score.cardinality = 5
column.score.totalDocs = 5
column.score.dataType = FLOAT
column.score.bitsPerElement = 3
column.score.lengthOfEachEntry = 0
column.score.columnType = METRIC
column.score.isSorted = false
column.score.hasDictionary = true
column.score.isSingleValues = true
column.score.maxNumberOfMultiValues = 0
column.score.totalNumberOfEntries = 5
column.score.isAutoGenerated = false
column.score.minValue = 3.0
column.score.maxValue = 3.8
column.score.defaultNullValue = 0.0
column.studentID.cardinality = 4
column.studentID.totalDocs = 5
column.studentID.dataType = INT
column.studentID.bitsPerElement = 2
column.studentID.lengthOfEachEntry = 0
column.studentID.columnType = DIMENSION
column.studentID.isSorted = true
column.studentID.hasDictionary = true
column.studentID.isSingleValues = true
column.studentID.maxNumberOfMultiValues = 0
column.studentID.totalNumberOfEntries = 5
column.studentID.isAutoGenerated = false
column.studentID.minValue = 200
column.studentID.maxValue = 203
column.studentID.defaultNullValue = -2147483648
column.subject.cardinality = 4
column.subject.totalDocs = 5
column.subject.dataType = STRING
column.subject.bitsPerElement = 2
column.subject.lengthOfEachEntry = 7
column.subject.columnType = DIMENSION
column.subject.isSorted = false
column.subject.hasDictionary = true
column.subject.isSingleValues = true
column.subject.maxNumberOfMultiValues = 0
column.subject.totalNumberOfEntries = 5
column.subject.isAutoGenerated = false
column.subject.minValue = English
column.subject.maxValue = Physics
column.subject.defaultNullValue = null
column.timestampInEpoch.cardinality = 5
column.timestampInEpoch.totalDocs = 5
column.timestampInEpoch.dataType = LONG
column.timestampInEpoch.bitsPerElement = 3
column.timestampInEpoch.lengthOfEachEntry = 0
column.timestampInEpoch.columnType = DATE_TIME
column.timestampInEpoch.isSorted = false
column.timestampInEpoch.hasDictionary = true
column.timestampInEpoch.isSingleValues = true
column.timestampInEpoch.maxNumberOfMultiValues = 0
column.timestampInEpoch.totalNumberOfEntries = 5
column.timestampInEpoch.isAutoGenerated = false
column.timestampInEpoch.datetimeFormat = 1:MILLISECONDS:EPOCH
column.timestampInEpoch.datetimeGranularity = 1:MILLISECONDS
column.timestampInEpoch.minValue = 1570863600000
column.timestampInEpoch.maxValue = 1572418800000
column.timestampInEpoch.defaultNullValue = -9223372036854775808
segment.index.version = v3

also have you logged to pinot server container and checked that pinot server loaded the segment successfully ?

also, can you try to change the ingestion job type to

"jobType" : "SegmentCreationAndTarPush",

wanna double check it’s not s3 credential issue or something else

similar issue reported in the past: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1687512488422419 do you by any chance have a successfully running environment, i.e. using

SegmentCreationAndMetadataPush

, with input/outputs all in s3? so that i can compare my configurations against it.. this bug is really stubborn i literally tried all i can think of that is possibly related 😂 (just guessing, i sense this might be deep store configuration issue..)

how do you configure the deep store in controller ?

we are using helm charts, controller conf:

data:
      dir: s3://<bucket name>

    extra:
      configs: |-
        controller.access.protocols.http.port=9000
        controller.enable.split.commit=true
        controller.local.temp.dir=/tmp/pinot-tmp-data
        controller.task.scheduler.enabled=true
        pinot.controller.segment.fetcher.protocols=file,http,s3
        pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
        pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
        pinot.controller.storage.factory.s3.disableAcl=false
        pinot.controller.storage.factory.s3.region=us-west-2
        pinot.set.instance.id.to.hostname=true

server conf:

extra:
  configs: |-
    pinot.controller.storage.factory.s3.disableAcl=false
    pinot.query.runner.port=7732
    pinot.query.server.port=7321
    pinot.server.instance.enable.split.commit=true
    pinot.server.instance.realtime.alloc.offheap=true
    pinot.server.instance.segment.store.uri=s3://<bucket name>
    pinot.server.segment.fetcher.protocols=file,http,s3
    pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
    pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
    pinot.server.storage.factory.s3.disableAcl=false
    pinot.server.storage.factory.s3.region=us-west-2
    pinot.set.instance.id.to.hostname=true

for segment metadata push, controller doesn’t touch the actual segment file, only segment push job does.

also if I remember correctly, the download file url should be

s3://<bucket name>/transcript/transcript_OFFLINE_1570863600000_1572418800000_0.tar.gz

is the error occur for new segment push or it's for existing table refresh

I think you need to check on pinot server data directory, if the local segment metadata contains the forward index

forward index is also enabled

With

"jobType" : "SegmentCreationAndMetadataPush"

there are open threads where people called out that they have faced similar issues but I nowhere found the solution for this Only way is to use

"jobType" : "SegmentCreationAndTarPush"

@Xiang Fu Extremely sorry for the delay, please find the pinot table and schema configs, job spec and server logs index_map contents

attribute_key.dictionary.startOffset = 0
attribute_key.dictionary.size = 81728
attribute_key.forward_index.startOffset = 81728
attribute_key.forward_index.size = 628387
attribute_value.dictionary.startOffset = 710115
attribute_value.dictionary.size = 2951371
attribute_value.forward_index.startOffset = 3661486
attribute_value.forward_index.size = 837846
event_name.dictionary.startOffset = 4499332
event_name.dictionary.size = 199673
event_name.forward_index.startOffset = 4699005
event_name.forward_index.size = 628387
frequency.dictionary.startOffset = 5327392
frequency.dictionary.size = 443500
frequency.forward_index.startOffset = 5770892
frequency.forward_index.size = 890211

Server Logs:

2025-08-08T12:27:35.807+05:30 88 START: CallbackHandler 0, INVOKE /pinot/INSTANCES/Server/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@248b8501 type: CALLBACK
2025-08-08T12:27:35.807+05:30 CallbackHandler 0 subscribing changes listener to path: /pinot/INSTANCES/Server/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@248b8501, watchChild: false
2025-08-08T12:27:35.809+05:30 CallbackHandler0, Subscribing to path: /pinot/INSTANCES/Server/MESSAGES took: 2
2025-08-08T12:27:35.817+05:30 The latency of message b5d65660-0a1c-4ccc-9cb4-accb539a32b0 is 51 ms
2025-08-08T12:27:35.852+05:30 No config found at /pinot/CONFIGS/RESOURCE/ems_attributes_metadata_push_OFFLINE.
2025-08-08T12:27:35.852+05:30 Scheduling message b5d65660-0a1c-4ccc-9cb4-accb539a32b0: ems_attributes_metadata_push_OFFLINE:ems_attributes_metadata_push_OFFLINE__0, OFFLINE->ONLINE
2025-08-08T12:27:35.853+05:30 Submit task: b5d65660-0a1c-4ccc-9cb4-accb539a32b0 to pool: java.util.concurrent.ThreadPoolExecutor@33b35491[Running, pool size = 40, active threads = 0, queued tasks = 0, completed tasks = 87]
2025-08-08T12:27:35.853+05:30 Message: b5d65660-0a1c-4ccc-9cb4-accb539a32b0 handling task scheduled
2025-08-08T12:27:35.853+05:30 88 END:INVOKE CallbackHandler 0, /pinot/INSTANCES/Server/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@248b8501 type: CALLBACK Took: 46ms
2025-08-08T12:27:35.853+05:30 handling task: b5d65660-0a1c-4ccc-9cb4-accb539a32b0 begin, at: 1754636255853
2025-08-08T12:27:35.853+05:30 handling message: b5d65660-0a1c-4ccc-9cb4-accb539a32b0 transit ems_attributes_metadata_push_OFFLINE.ems_attributes_metadata_push_OFFLINE__0|[] from:OFFLINE to:ONLINE, relayedFrom: null
2025-08-08T12:27:35.853+05:30 Instance Server, partition ems_attributes_metadata_push_OFFLINE__0 received state transition from OFFLINE to ONLINE on session 3000001d64900bf, message id: b5d65660-0a1c-4ccc-9cb4-accb539a32b0
2025-08-08T12:27:35.853+05:30 SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline() : ZnRecord=b5d65660-0a1c-4ccc-9cb4-accb539a32b0, {CREATE_TIMESTAMP=1754636255766, ClusterEventName=IdealStateChange, EXECUTE_START_TIMESTAMP=1754636255853, EXE_SESSION_ID=3000001d64900bf, FROM_STATE=OFFLINE, MSG_ID=b5d65660-0a1c-4ccc-9cb4-accb539a32b0, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=ems_attributes_metadata_push_OFFLINE__0, READ_TIMESTAMP=1754636255817, RESOURCE_NAME=ems_attributes_metadata_push_OFFLINE, RESOURCE_TAG=ems_attributes_metadata_push_OFFLINE, RETRY_COUNT=3, SRC_NAME=pinot-v2-controller-2.pinot-v2-controller-headless.dataplatform.svc.cluster.local_9000, SRC_SESSION_ID=1000001adf200bf, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server, TGT_SESSION_ID=3000001d64900bf, TO_STATE=ONLINE}{}{}, Stat=Stat {_version=0, _creationTime=1754636255800, _modifiedTime=1754636255800, _ephemeralOwner=0}
2025-08-08T12:27:35.853+05:30 Creating table data manager for table: ems_attributes_metadata_push_OFFLINE
2025-08-08T12:27:35.858+05:30 Initializing table data manager for table: ems_attributes_metadata_push_OFFLINE
2025-08-08T12:27:35.858+05:30 Initialized table data manager with data directory: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE
2025-08-08T12:27:35.858+05:30 Starting table data manager
2025-08-08T12:27:35.858+05:30 Started table data manager
2025-08-08T12:27:35.858+05:30 Created table data manager for table: ems_attributes_metadata_push_OFFLINE
2025-08-08T12:27:35.858+05:30 Adding ONLINE segment: ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:35.866+05:30 Adding new ONLINE segment: ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:35.867+05:30 Segment: ems_attributes_metadata_push_OFFLINE__0 does not exist
2025-08-08T12:27:35.867+05:30 Downloading and loading segment: ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:35.867+05:30 Downloading segment: ems_attributes_metadata_push_OFFLINE__0 from: gs://{DEEPSTORE_BUCKET}/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:35.867+05:30 Copy gs://{DEEPSTORE_BUCKET}/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0 to local /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0.tar.compressed
2025-08-08T12:27:36.053+05:30 Fetched segment from: gs://{DEEPSTORE_BUCKET}/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0 to: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0.tar.compressed of size: 1027
2025-08-08T12:27:36.053+05:30 Downloaded tarred segment: ems_attributes_metadata_push_OFFLINE__0 from: gs://{DEEPSTORE_BUCKET}/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0 to: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0.tar.compressed, file length: 1027
2025-08-08T12:27:36.053+05:30 Untarring segment: ems_attributes_metadata_push_OFFLINE__0 from: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0.tar.compressed to: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:36.053+05:30 Untarred segment: ems_attributes_metadata_push_OFFLINE__0 into: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0/segmentMetadataDir-db8da2a8-8c23-46cb-90b0-3682b6491c31
2025-08-08T12:27:36.053+05:30 Downloaded segment: ems_attributes_metadata_push_OFFLINE__0 from: gs://{DEEPSTORE_BUCKET}/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0 to: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:36.058+05:30 Trying to enable the forward index for a column: attribute_key of segment: ems_attributes_metadata_push_OFFLINE__0 missing either the dictionary (disabled) and / or the inverted index (disabled) is not possible. Either a refresh or back-fill is required to get the forward index, ignoring
2025-08-08T12:27:36.058+05:30 Trying to enable the forward index for a column: attribute_value of segment: ems_attributes_metadata_push_OFFLINE__0 missing either the dictionary (disabled) and / or the inverted index (disabled) is not possible. Either a refresh or back-fill is required to get the forward index, ignoring
2025-08-08T12:27:36.058+05:30 Trying to enable the forward index for a column: event_name of segment: ems_attributes_metadata_push_OFFLINE__0 missing either the dictionary (disabled) and / or the inverted index (disabled) is not possible. Either a refresh or back-fill is required to get the forward index, ignoring
2025-08-08T12:27:36.058+05:30 Trying to enable the forward index for a column: frequency of segment: ems_attributes_metadata_push_OFFLINE__0 missing either the dictionary (disabled) and / or the inverted index (disabled) is not possible. Either a refresh or back-fill is required to get the forward index, ignoring
2025-08-08T12:27:36.061+05:30 Successfully loaded segment: ems_attributes_metadata_push_OFFLINE__0 with SegmentDirectory
2025-08-08T12:27:36.061+05:30 Adding immutable segment: ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:36.061+05:30 Added new immutable segment: ems_attributes_metadata_push_OFFLINE__0
2025-08-08T12:27:36.061+05:30 Downloaded and loaded segment: ems_attributes_metadata_push_OFFLINE__0 with CRC: 4264216006 on tier: default
2025-08-08T12:27:36.067+05:30 Message b5d65660-0a1c-4ccc-9cb4-accb539a32b0 completed.
2025-08-08T12:27:36.070+05:30 Delete message b5d65660-0a1c-4ccc-9cb4-accb539a32b0 from zk!
2025-08-08T12:27:36.071+05:30 message finished: b5d65660-0a1c-4ccc-9cb4-accb539a32b0, took 217
2025-08-08T12:27:36.071+05:30 Message: b5d65660-0a1c-4ccc-9cb4-accb539a32b0 (parent: null) handling task for ems_attributes_metadata_push_OFFLINE:ems_attributes_metadata_push_OFFLINE__0 completed at: 1754636256071, results: true. FrameworkTime: 4 ms; HandlerTime: 214 ms.
2025-08-08T12:27:36.074+05:30 88 START: CallbackHandler 0, INVOKE /pinot/INSTANCES/Server/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@248b8501 type: CALLBACK
2025-08-08T12:27:36.074+05:30 CallbackHandler 0 subscribing changes listener to path: /pinot/INSTANCES/Server/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@248b8501, watchChild: false
2025-08-08T12:27:36.076+05:30 CallbackHandler0, Subscribing to path: /pinot/INSTANCES/Server/MESSAGES took: 2
2025-08-08T12:27:36.078+05:30 No Messages to process
2025-08-08T12:27:36.078+05:30 88 END:INVOKE CallbackHandler 0, /pinot/INSTANCES/Server/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@248b8501 type: CALLBACK Took: 4ms

Spark Job Logs:

25/08/08 07:29:56 INFO PinotFSFactory: Initializing PinotFS for scheme gs, classname org.apache.pinot.plugin.filesystem.GcsPinotFS
25/08/08 07:29:57 INFO GcsPinotFS: Configs using default credential
25/08/08 07:29:57 WARN SparkSegmentGenerationJobRunner: Cannot find local Pinot plugins tar file at [{HADDOP_YARN_LOCAL_PATH}/pinot-plugins.tar.gz]
25/08/08 07:29:57 INFO SparkSegmentGenerationJobRunner: Trying to copy input file from gs://{GCS_BUCKET}/warehouse/dataengineering.db/ems_attributes/data/20250807_215857_08174_e5spa-547e36f6-4d81-49e2-a85a-a4bee3689c15.parquet to {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/input/20250807_215857_08174_e5spa-547e36f6-4d81-49e2-a85a-a4bee3689c15.parquet
25/08/08 07:29:57 INFO GcsPinotFS: Copy gs://{GCS_BUCKET}/warehouse/dataengineering.db/ems_attributes/data/20250807_215857_08174_e5spa-547e36f6-4d81-49e2-a85a-a4bee3689c15.parquet to local {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/input/20250807_215857_08174_e5spa-547e36f6-4d81-49e2-a85a-a4bee3689c15.parquet
25/08/08 07:29:58 WARN SegmentIndexCreationDriverImpl: Using class: org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader to read segment, ignoring configured file format: AVRO
25/08/08 07:29:58 INFO CodecPool: Got brand-new decompressor [.zstd]
25/08/08 07:29:59 INFO CodecPool: Got brand-new decompressor [.zstd]
25/08/08 07:30:01 INFO SegmentIndexCreationDriverImpl: Finished building StatsCollector!
25/08/08 07:30:01 INFO SegmentIndexCreationDriverImpl: Collected stats for 418919 documents
25/08/08 07:30:01 INFO Jvm: Chronicle core loaded from file:{HADDOP_YARN_LOCAL_PATH}/pinot-all-1.3.0-jar-with-dependencies.jar
25/08/08 07:30:01 INFO PinotDataBuffer: Using Unsafe as buffer on JVM version 11
25/08/08 07:30:01 INFO PinotDataBuffer: Instantiating Pinot buffer factory class org.apache.pinot.segment.spi.memory.unsafe.UnsafePinotBufferFactory
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Using fixed length dictionary for column: attribute_key, size: 81720
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Created dictionary for STRING column: attribute_key with cardinality: 2270, max length in bytes: 36, range:  to year
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Using fixed length dictionary for column: event_name, size: 199665
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Created dictionary for STRING column: event_name with cardinality: 3915, max length in bytes: 51, range:  to yes_bank_closure
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Using fixed length dictionary for column: attribute_value, size: 2951363
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Created dictionary for STRING column: attribute_value with cardinality: 48383, max length in bytes: 61, range: 
 to 
25/08/08 07:30:01 INFO SegmentDictionaryCreator: Created dictionary for INT column: frequency with cardinality: 110873, range: -2145165550 to 2144703085
25/08/08 07:30:01 INFO SegmentColumnarIndexCreator: Column: attribute_key is not nullable
25/08/08 07:30:01 INFO SegmentColumnarIndexCreator: Column: attribute_value is not nullable
25/08/08 07:30:01 INFO SegmentColumnarIndexCreator: Column: event_name is not nullable
25/08/08 07:30:01 INFO SegmentColumnarIndexCreator: Column: frequency is not nullable
25/08/08 07:30:01 INFO CodecPool: Got brand-new decompressor [.zstd]
25/08/08 07:30:01 INFO SegmentIndexCreationDriverImpl: Start building IndexCreator!
25/08/08 07:30:02 INFO SegmentIndexCreationDriverImpl: Finished records indexing in IndexCreator!
25/08/08 07:30:02 INFO SegmentIndexCreationDriverImpl: Finished segment seal!
25/08/08 07:30:02 INFO SegmentV1V2ToV3FormatConverter: Converting segment: {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0 to v3 format
25/08/08 07:30:03 INFO Reflections: Reflections took 423 ms to scan 1 urls, producing 17 keys and 41 values
25/08/08 07:30:03 INFO SegmentDirectoryLoaderRegistry: Initialized SegmentDirectoryLoaderRegistry with 2 segmentDirectoryLoaders: [default, tierBased] in 460 ms
25/08/08 07:30:03 INFO SegmentV1V2ToV3FormatConverter: v3 segment location for segment: ems_attributes_metadata_push_OFFLINE__0 is {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0/v3
25/08/08 07:30:03 INFO SegmentV1V2ToV3FormatConverter: Deleting files in v1 segment directory: {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0
25/08/08 07:30:03 INFO CrcUtils: Computed crc = 4264216006, based on files [{HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0/v3/columns.psf, {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0/v3/index_map, {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0/v3/metadata.properties]
25/08/08 07:30:03 INFO SegmentIndexCreationDriverImpl: Driver, record read time (in ms) : 209
25/08/08 07:30:03 INFO SegmentIndexCreationDriverImpl: Driver, stats collector time (in ms) : 1773
25/08/08 07:30:03 INFO SegmentIndexCreationDriverImpl: Driver, indexing time (in ms) : 316
25/08/08 07:30:03 INFO SparkSegmentGenerationJobRunner: Tarring segment from: {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0 to: {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0.tar.gz
25/08/08 07:30:03 INFO SparkSegmentGenerationJobRunner: Size for segment: ems_attributes_metadata_push_OFFLINE__0, uncompressed: 6.36M, compressed: 2.09M
25/08/08 07:30:03 INFO SegmentGenerationJobUtils: Trying to move metadata tar file from: [{HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0.tar.gz] to [gs://{DEEPSTORE_BUCKET}/pinot-batch-ingestion/staging/ems_attributes_metadata_push_OFFLINE__0.tar.gz]
25/08/08 07:30:03 INFO GcsPinotFS: Copying file {HADDOP_YARN_LOCAL_PATH}/tmp/pinot-205ff255-37f3-463f-bc24-7ad4227de069/output/ems_attributes_metadata_push_OFFLINE__0.tar.gz to uri gs://{DEEPSTORE_BUCKET}/pinot-batch-ingestion/staging/ems_attributes_metadata_push_OFFLINE__0.tar.gz
25/08/08 07:30:04 INFO PinotFSFactory: Initializing PinotFS for scheme gs, classname org.apache.pinot.plugin.filesystem.GcsPinotFS
25/08/08 07:30:04 INFO GcsPinotFS: Configs using default credential
25/08/08 07:30:05 INFO SegmentPushUtils: Start pushing segment metadata: {gs://{DEEPSTORE_BUCKET}/pinot-segments/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0.tar.gz=gs://{DEEPSTORE_BUCKET}/pinot-segments/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0.tar.gz} to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@3f0c56c9] for table ems_attributes_metadata_push
25/08/08 07:30:05 INFO SegmentPushUtils: Checking if metadata tar gz file gs://{DEEPSTORE_BUCKET}/pinot-segments/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0.metadata.tar.gz exists
25/08/08 07:30:05 INFO GcsPinotFS: Copy gs://{DEEPSTORE_BUCKET}/pinot-segments/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0.tar.gz to local {HADDOP_YARN_LOCAL_PATH}/tmp/segmentTar-5a36c1ff-a3be-48ea-a55d-95d321968efe.tar.gz
25/08/08 07:30:05 INFO SegmentPushUtils: Trying to untar Metadata file from: [{HADDOP_YARN_LOCAL_PATH}/tmp/segmentTar-5a36c1ff-a3be-48ea-a55d-95d321968efe.tar.gz] to [{HADDOP_YARN_LOCAL_PATH}/tmp/segmentMetadataDir-5a36c1ff-a3be-48ea-a55d-95d321968efe]
25/08/08 07:30:05 INFO SegmentPushUtils: Trying to untar CreationMeta file from: [{HADDOP_YARN_LOCAL_PATH}/tmp/segmentTar-5a36c1ff-a3be-48ea-a55d-95d321968efe.tar.gz] to [{HADDOP_YARN_LOCAL_PATH}/tmp/segmentMetadataDir-5a36c1ff-a3be-48ea-a55d-95d321968efe]
25/08/08 07:30:05 INFO SegmentPushUtils: Trying to tar segment metadata dir [{HADDOP_YARN_LOCAL_PATH}/tmp/segmentMetadataDir-5a36c1ff-a3be-48ea-a55d-95d321968efe] to [{HADDOP_YARN_LOCAL_PATH}/tmp/segmentMetadata-5a36c1ff-a3be-48ea-a55d-95d321968efe.tar.gz]
25/08/08 07:30:05 INFO SegmentPushUtils: Pushing segment: ems_attributes_metadata_push_OFFLINE__0 to location: {CONTROLLER_URL} for table ems_attributes_metadata_push
25/08/08 07:30:05 INFO HttpClient: Sending request: /v2/segments?tableName=ems_attributes_metadata_push to controller: pinot-v2-controller-0.pinot-v2-controller-headless.dataplatform.svc.cluster.local, version: Unknown
25/08/08 07:30:05 INFO SegmentPushUtils: Response for pushing table ems_attributes_metadata_push segment ems_attributes_metadata_push_OFFLINE__0 to location {CONTROLLER_URL} - 200: {"status":"Successfully uploaded segment: ems_attributes_metadata_push_OFFLINE__0 of table: ems_attributes_metadata_push_OFFLINE"}

This is weird

2025-08-08T12:27:36.053+05:30 Downloaded tarred segment: ems_attributes_metadata_push_OFFLINE__0 from: gs://{DEEPSTORE_BUCKET}/ems_attributes_metadata_push/ems_attributes_metadata_push_OFFLINE__0 to: /var/pinot/server/data/index/ems_attributes_metadata_push_OFFLINE/tmp/tmp-ems_attributes_metadata_push_OFFLINE__0-44088095-a053-451a-83bc-27615f5368f5/ems_attributes_metadata_push_OFFLINE__0.tar.compressed, file length: 1027

why the segment size becomes this small

what's your controller config?

Controller Config:

# ------------------------------------------------------------------------------
  # Pinot Controller:
  # ------------------------------------------------------------------------------
  controller:
    name: controller
    replicaCount: 3
    podManagementPolicy: Parallel
    podSecurityContext: {}
      # fsGroup: 2000
    securityContext: {}
    startCommand: "StartController"

    probes:
      endpoint: "/health"
      livenessEnabled: false
      readinessEnabled: false
      startupEnabled: false
      liveness:
        initialDelaySeconds: 60
        failureThreshold: 10
        timeoutSeconds: 10
        successThreshold: 1
        periodSeconds: 10
  
      readiness:
        initialDelaySeconds: 60
        failureThreshold: 10
        timeoutSeconds: 10
        successThreshold: 1
        periodSeconds: 10
  
      startup:
        initialDelaySeconds: 60
        failureThreshold: 10
        timeoutSeconds: 10
        successThreshold: 1
        periodSeconds: 10

    serviceMonitor:
      enabled: true
      namespace: "dataplatform"
      ScrapeInterval: "60s"
      ScrapeTimeout: "60s"
      additionalLabels: 
        release: prometheus

    persistence:
      enabled: true
      accessMode: ReadWriteOnce
      size: 30G
      mountPath: /var/pinot/controller/data
      # storageClass: ""
      storageClass: "fast-pd-standard-xfs"
      extraVolumes: []
      extraVolumeMounts: []

    data:
      dir: gs://{DEEPSTORE_BUCKET}

    vip:
      enabled: false
      host: pinot-controller
      port: 9000

    jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml -XX:ActiveProcessorCount=2 -Djute.maxbuffer=4194304 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log"

    log4j2ConfFile: /opt/pinot/conf/log4j2.xml
    pluginsDir: /opt/pinot/plugins

    pdb:
      enabled: false 
      minAvailable: ""
      maxUnavailable: 50%

    service:
      annotations: {}
      clusterIP: "None"
      externalIPs: []
      loadBalancerIP: ""
      loadBalancerSourceRanges: []
      type: ClusterIP
      port: 9000
      nodePort: ""
      protocol: TCP
      name: controller
      metricsport: 8008
      metricsname: jmx-metric

    external:
      enabled: false
      type: LoadBalancer
      port: 9000
      annotations: {}

    ingress:
      v1beta1:
        enabled: false
        annotations: { }
        tls: { }
        path: /
        hosts: [ ]
      v1:
        enabled: true
        ingressClassName: "internal-nginx-gateway"
        annotations: 
          <http://nginx.ingress.kubernetes.io/use-regex|nginx.ingress.kubernetes.io/use-regex>: "true"
          <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: "128m"
        tls: 
          - hosts:
              - {CONTROLLER_URL}
            secretName: {SECRETS}
        path: /
        hosts: 
          - {CONTROLLER_URL}

    resources:
      requests:
        cpu: 4
        memory: "8Gi"
      limits:
        cpu: 4
        memory: "8Gi"

    nodeSelector:
      pool-type: prod-dp-spot-pinot

    tolerations:
      - key: dp-node-pinot
        operator: Equal
        value: 'true'
        effect: NoSchedule

    affinity: {}

    podAnnotations: {}
    automaticReload:
      enabled: false

    updateStrategy:
      type: RollingUpdate

    # Use envFrom to define all of the ConfigMap or Secret data as container environment variables.
    # ref: <https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables>
    # ref: <https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables>
    envFrom: []
    #  - configMapRef:
    #      name: special-config
    #  - secretRef:
    #      name: test-secret

    # Use extraEnv to add individual key value pairs as container environment variables.
    # ref: <https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/>
    extraEnv:
      - name: LOG4J_CONSOLE_LEVEL
        value: info
    #  - name: PINOT_CUSTOM_ENV
    #    value: custom-value

    # Extra configs will be appended to pinot-controller.conf file
    extra:
      configs: |-
        controller.local.temp.dir=/var/pinot/controller/data
        controller.task.scheduler.enabled=true
        pinot.set.instance.id.to.hostname=true
        pinot.controller.segment.fetcher.protocols=file,http,gs
        pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
        <http://pinot.controller.storage.factory.class.gs|pinot.controller.storage.factory.class.gs>=org.apache.pinot.plugin.filesystem.GcsPinotFS
        pinot.controller.storage.factory.gs.projectId=gw-backend-production
        controller.disable.ingestion.groovy=false
        controller.deleted.segments.retentionInDays=2

@Xiang Fu please let me know if you need anything else on this

and we have the untarred segment files stored on the deepstore bucket