Emerson Lesage
07/30/2025, 3:48 PMMayank
Mayank
Emerson Lesage
07/30/2025, 4:27 PM"thread":"PinotHelixResourceManagerExecutorService","level":"WARN","loggerName":"org.apache.pinot.controller.helix.core.SegmentDeletionManager","message":"Failed to delete segment table_OFFLINE_0_5c7bcf7a-4a30-4be2-851e-cda64f6634f2 from file:/var/pinot/controller/data/table/table_OFFLINE_0_5c7bcf7a-4a30-4be2-851e-cda64f6634f2"
"thread":"PinotHelixResourceManagerExecutorService","level":"INFO","loggerName":"org.apache.pinot.controller.helix.core.SegmentDeletionManager","message":"Deleted 1 segments from table table_OFFLINE:[table_OFFLINE_0_5c7bcf7a-4a30-4be2-851e-cda64f6634f2]"Emerson Lesage
07/30/2025, 4:29 PMFailed to find local segment file for segment file:/var/pinot/controller/data/table/table_OFFLINE_1751068800000_1751068800000_1_e8b20b3b-5af7-49e7-8954-47b38cde10e4Mayank
Emerson Lesage
07/30/2025, 6:55 PM# ------------------------------------------------------------------------------
# Pinot Controller:
# ------------------------------------------------------------------------------
controller:
name: controller
replicaCount: 1
podManagementPolicy: Parallel
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
startCommand: "StartController"
...
data:
dir: /var/pinot/controller/data
...Emerson Lesage
07/30/2025, 6:57 PMMayank
Emerson Lesage
07/30/2025, 7:00 PMEmerson Lesage
07/30/2025, 7:04 PM# Extra configs will be appended to pinot-controller.conf file
extra:
configs: |-
pinot.controller.metrics.enabled=true
pinot.controller.metrics.enablePercentileMetrics=true
pinot.controller.logging.level=DEBUG
pinot.set.instance.id.to.hostname=true
controller.task.scheduler.enabled=true
pinot.controller.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.controller.storage.factory.gs.projectId=project
pinot.controller.storage.factory.gs.gcpKey=/vault/secrets/pinot-hyperion-sa
pinot.controller.segment.fetcher.protocols=file,http,gs,s3
pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-east-1
pinot.controller.storage.dataDir=<s3://bucket>
pinot.ingestion.batchSize=1000
pinot.ingestion.numThreads=4
pinot.ingestion.timeout=300000
pinot.server.ingestion.maxBatchSize=2000
pinot.server.ingestion.numThreads=4
pinot.server.ingestion.timeout=300000Frank Danielson
07/30/2025, 9:30 PMEmerson Lesage
07/31/2025, 1:13 PMcontroller.data.dir=<s3://bucket>
and
controller.local.temp.dir=/some/local/path ?
or is their some other config we should be setting? We are currently using minions and the SegmentGenerationAndPushTask to do ingestion and set outputDirURI in the job config.Mayank
Emerson Lesage
07/31/2025, 1:27 PM"localizedMessage": "Local temporary directory is not configured, cannot use remote data directory: <s3://bucket> as local temporary directory",
"message": "Local temporary directory is not configured, cannot use remote data directory: <s3://bucket> as local temporary directory",
"name": "java.lang.IllegalStateException",Mayank
Frank Danielson
07/31/2025, 5:33 PMMayank
pinot.controller.dataDir. I am guessing the way you are invoking Pinot (with helm/k8s) is causing the issue. There's nothing called values.controller.data.dir in the code base, I think that is a helm setting?Frank Danielson
07/31/2025, 5:37 PM/var/pinot/controller/dataFrank Danielson
07/31/2025, 5:41 PMdata:
pinot-controller.conf: |-
controller.data.dir={{ .Values.controller.data.dir }}Mayank
Frank Danielson
07/31/2025, 5:44 PM"Local temporary directory is not
configured, cannot use remote data directory: <s3://bucket/path> as local temporary directory","message":"Local temporary directory is not configured, cannot use remote data directory: <s3://bucket/path> as local temporary directory"Frank Danielson
07/31/2025, 5:45 PMpinot.controller.dataDir=<s3://bucket/path>Mayank
Mayank
controller.data.dir -> should point to deepstore
• controller.local.temp.dir -> should point to local directoryFrank Danielson
07/31/2025, 5:49 PM/var/pinot/controller/data
Same directory as the values.data.dir as a test
So you're suggesting I set controller.local.temp.dir in the extra config to something like /var/pinot/controller/data, and change the values.controller.data.dir in helm to the S3 bucket?Mayank