Hello, We have our deepstore set up with s3 and w...
# troubleshooting
e
Hello, We have our deepstore set up with s3 and was wondering if calling the delete segments endpoint deletes the segment from the table, as well as s3. Thanks!
m
Yes it does.
There’s a deleted segment retention config to keep it around for sometime
e
Ah, we were seeing some issues where segments are properly being deleted from a table, but not our deepstore. We are seeing some errors in the controller logs pertaining to local files, but not sure if those are red herrings or not. Example:
Copy code
"thread":"PinotHelixResourceManagerExecutorService","level":"WARN","loggerName":"org.apache.pinot.controller.helix.core.SegmentDeletionManager","message":"Failed to delete segment table_OFFLINE_0_5c7bcf7a-4a30-4be2-851e-cda64f6634f2 from file:/var/pinot/controller/data/table/table_OFFLINE_0_5c7bcf7a-4a30-4be2-851e-cda64f6634f2"

"thread":"PinotHelixResourceManagerExecutorService","level":"INFO","loggerName":"org.apache.pinot.controller.helix.core.SegmentDeletionManager","message":"Deleted 1 segments from table table_OFFLINE:[table_OFFLINE_0_5c7bcf7a-4a30-4be2-851e-cda64f6634f2]"
We see the same sorts of local file errors when looking at the logs from the retention manager as well
Copy code
Failed to find local segment file for segment file:/var/pinot/controller/data/table/table_OFFLINE_1751068800000_1751068800000_1_e8b20b3b-5af7-49e7-8954-47b38cde10e4
m
It should try to delete file on deepstore. What config points to /var/pinot/controller/data in your setup? That seems incorrect.
e
that is in our values.yaml for our controller:
Copy code
# ------------------------------------------------------------------------------
# Pinot Controller:
# ------------------------------------------------------------------------------
controller:
  name: controller
  replicaCount: 1
  podManagementPolicy: Parallel
  podSecurityContext: {}
    # fsGroup: 2000
  securityContext: {}
  startCommand: "StartController"

...

  data:
    dir: /var/pinot/controller/data

...
should be the default i believe
m
That is not deepstore, it is pointing to local disk on controller
e
Ah so that needs to point to our s3 location?
We also have these extras in our config:
Copy code
# Extra configs will be appended to pinot-controller.conf file
  extra:
    configs: |-
      pinot.controller.metrics.enabled=true
      pinot.controller.metrics.enablePercentileMetrics=true
      pinot.controller.logging.level=DEBUG
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true
      pinot.controller.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
      pinot.controller.storage.factory.gs.projectId=project
      pinot.controller.storage.factory.gs.gcpKey=/vault/secrets/pinot-hyperion-sa
      pinot.controller.segment.fetcher.protocols=file,http,gs,s3
      pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.controller.storage.factory.s3.region=us-east-1
      pinot.controller.storage.dataDir=<s3://bucket>
      pinot.ingestion.batchSize=1000 
      pinot.ingestion.numThreads=4 
      pinot.ingestion.timeout=300000 
      pinot.server.ingestion.maxBatchSize=2000 
      pinot.server.ingestion.numThreads=4 
      pinot.server.ingestion.timeout=300000
f
Just subbing to this thread
e
@Mayank in order for the controller to properly delete segments from the deep store do we need to set
controller.data.dir=<s3://bucket>
and
controller.local.temp.dir=/some/local/path
? or is their some other config we should be setting? We are currently using minions and the
SegmentGenerationAndPushTask
to do ingestion and set
outputDirURI
in the job config.
m
Yes you need to set data dir to deepstore. Please check the docs on deep store setup
e
Ah ok and just to confirm if controller.data.dir points to an s3 path, then we have to explicitly set controller.local.temp.dir right? we were getting some of the following errors when trying to point to s3:
Copy code
"localizedMessage": "Local temporary directory is not configured, cannot use remote data directory: <s3://bucket> as local temporary directory",
        "message": "Local temporary directory is not configured, cannot use remote data directory: <s3://bucket> as local temporary directory",
        "name": "java.lang.IllegalStateException",
m
First config points to deepstore for storing all data. Tmp dir is for receiving / sending tarring/ untarring of temporary files
f
I feel like there's two configurations here, values.controller.data.dir is the one that errors if we set it to S3 and forces the controller in to a crashloop. The second is in the values.controller.extra.configs: pinot.controller.daraDir, which is pointed to S3.
m
As far as Pinot source code is concerned, there is only one
pinot.controller.dataDir
. I am guessing the way you are invoking Pinot (with helm/k8s) is causing the issue. There's nothing called
values.controller.data.dir
in the code base, I think that is a helm setting?
f
Yeah in the helm values.yaml defaults it's set to
/var/pinot/controller/data
Seems like it's referenced in the resulting configmap for the controller config:
Copy code
data:
  pinot-controller.conf: |-
    controller.data.dir={{ .Values.controller.data.dir }}
m
yeah, so I just passing along deepstore config there should work right?
f
Passing that an S3 bucket/path crashloops the pod with:
Copy code
"Local temporary directory is not 
configured, cannot use remote data directory: <s3://bucket/path> as local temporary directory","message":"Local temporary directory is not configured, cannot use remote data directory: <s3://bucket/path> as local temporary directory"
I did set this
Copy code
pinot.controller.dataDir=<s3://bucket/path>
m
and what did you set for local?
If you meant there are two different ones like below, then you are right: •
controller.data.dir
-> should point to deepstore •
controller.local.temp.dir
-> should point to local directory
f
We tried to set that with `values.controller.local.temp.dir`:
/var/pinot/controller/data
Same directory as the
values.data.dir
as a test So you're suggesting I set
controller.local.temp.dir
in the extra config to something like
/var/pinot/controller/data
, and change the
values.controller.data.dir
in helm to the S3 bucket?
m
I am suggesting that however you wire the values.yaml, in the end it should get these values: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1753984023684059?thread_ts=1753890499.367689&amp;cid=C011C9JHN7R
👀 1