Sota
05/14/2025, 10:34 PMbash-5.2# curl -X POST -F file=@data.json -H "Content-Type: multipart/form-data" -H "Authorization: Basic YW...Zo" "localhost:9000/ingestFromFile?tableNameWithType=test9_OFFLINE&batchConfigMapStr=%7B%22inputFormat%22%3A%22json%22%7D"
{"code":500,"error":"Caught exception when ingesting file into table: test9_OFFLINE. Caught exception while uploading segments. Push mode: TAR, segment tars: [[file:/tmp/data/ingestion_dir/working_dir_test9_OFFLINE_1747261509464_CLxUdOoBaw/segment_tar_dir/test9_1735676097000_1735676097000_1747261509468.tar.gz]]"}
in controller logs:
Caused by: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for CREATE '/v2/segments' for table 'test9_OFFLINE'" while sending request: /v2/segments?tableName=test9_OFFLINE&tableType=OFFLINE&tableName=test9_OFFLINE&tableType=OFFLINE to controller: pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown
I believe I've followed everything in the auth docs.Xiang Fu
Xiang Fu
Sota
05/15/2025, 2:47 PMSota
05/15/2025, 2:50 PMbash-5.2# find /tmp/data/ingestion_dir/
/tmp/data/ingestion_dir/Sota
05/15/2025, 4:07 PMXiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
Sota
05/15/2025, 11:43 PMaws s3 cp data.json s3://path/to/data/tmp/
in query console:
SET taskName = 'test';
SET input.fs.className = 'org.apache.pinot.plugin.filesystem.S3PinotFS';
INSERT INTO "test9"
FROM FILE 's3://path/to/data/tmp/'
failed:
Error Code: 200
QueryExecutionError:
org.apache.hc.core5.http.HttpException: Unable to get tasks states map. Error code 415, Error message: {"code":415,"error":"HTTP 415 Unsupported Media Type"}
at org.apache.pinot.common.minion.MinionClient.executeTask(MinionClient.java:130)
at org.apache.pinot.core.query.executor.sql.SqlQueryExecutor.executeDMLStatement(SqlQueryExecutor.java:100)
at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:191)
at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:125)
The pod the minion is running on has aws s3 permissions from its service account.Sota
05/16/2025, 12:41 AM"ingestionConfig": {
"batchIngestionConfig": {
"segmentIngestionType": "APPEND",
"segmentIngestionFrequency": "DAILY",
"batchConfigMaps": [
{
"input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
"input.fs.prop.region": "us-west-2",
"inputDirURI": "s3://path/to/data/tmp/",
"includeFileNamePattern": "glob:**/*.json",
"excludeFileNamePattern": "glob:**/*.tmp",
"inputFormat": "json"
}
]
}
},
"task": {
"taskTypeConfigsMap": {
"SegmentGenerationAndPushTask": {
"schedule": "0 * * * * ?",
"tableMaxNumTasks": "10"
}
}
}
and see it's able to list the files from s3:
Execute CronJob: table - test9_OFFLINE, task - SegmentGenerationAndPushTask at 2025-05-16T00:40:00.001+0000
Trying to schedule task type: SegmentGenerationAndPushTask, isLeader: false
Trying to extract input files from path: s3://path/to/data/tmp/, and exclude input files from existing segments metadata: [s3://path/to/data/tmp/data.json], and input files from running tasks: []
Listed 1 files from URI: s3://path/to/data/tmp/, is recursive: true
Final input files for task config generation: []
Finished CronJob: table - test9_OFFLINE, task - SegmentGenerationAndPushTask, next runtime is 2025-05-16T00:41:00.000+000Sota
05/16/2025, 12:41 AMXiang Fu
Sota
05/16/2025, 6:07 PM./bin/pinot-admin.sh CreateSegment -> UploadSegment
./bin/pinot-admin.sh GenerateData -> ImportData
Download segment from s3 -> curl -X POST /segments
Download segment from s3 -> curl -X POST /v2/segments
The segments get uploaded and I see them on s3 fine, but the segment status is BAD
java.lang.reflect.InvocationTargetException: null
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.lang.UnsupportedOperationException
at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchUntarSegmentToLocalStreamed(BaseSegmentFetcher.java:109) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndStreamUntarToLocal(SegmentFetcherFactory.java:147) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndStreamUntarToLocal(SegmentFetcherFactory.java:154) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:786) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:746) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndLoadSegment(BaseTableDataManager.java:389) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.core.data.manager.BaseTableDataManager.addNewOnlineSegment(BaseTableDataManager.java:360) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.doAddOnlineSegment(OfflineTableDataManager.java:54) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.core.data.manager.BaseTableDataManager.addOnlineSegment(BaseTableDataManager.java:313) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOnlineSegment(HelixInstanceDataManager.java:275) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:131) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
... 12 moreSota
05/16/2025, 6:29 PMXiang Fu
Xiang Fu