Having some issues with basic auth: ```bash-5.2# c...
# troubleshooting
s
Having some issues with basic auth:
Copy code
bash-5.2# curl -X POST -F file=@data.json  -H "Content-Type: multipart/form-data" -H "Authorization: Basic YW...Zo"  "localhost:9000/ingestFromFile?tableNameWithType=test9_OFFLINE&batchConfigMapStr=%7B%22inputFormat%22%3A%22json%22%7D"

{"code":500,"error":"Caught exception when ingesting file into table: test9_OFFLINE. Caught exception while uploading segments. Push mode: TAR, segment tars: [[file:/tmp/data/ingestion_dir/working_dir_test9_OFFLINE_1747261509464_CLxUdOoBaw/segment_tar_dir/test9_1735676097000_1735676097000_1747261509468.tar.gz]]"}
in controller logs:
Copy code
Caused by: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 403 (Forbidden) with reason: "Permission is denied for CREATE '/v2/segments' for table 'test9_OFFLINE'" while sending request: /v2/segments?tableName=test9_OFFLINE&tableType=OFFLINE&tableName=test9_OFFLINE&tableType=OFFLINE to controller: pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown
I believe I've followed everything in the auth docs.
x
Can you try to upload the segment generated directly using curl and auth header?
Not sure if the segment uploader carries the auth or not. Need to check that
s
Is there a way to generate the segment manually? If I recall correctly, it generated the segment in a tmp dir then deleted it after it failed.
Copy code
bash-5.2# find /tmp/data/ingestion_dir/
/tmp/data/ingestion_dir/
Found this doc https://docs.pinot.apache.org/users/tutorials/create-pinot-segments#creating-pinot-segments-outside-of-hadoop Just to clarify - is the /v2/segments api endpoint compatible with the segments generated via pinot-admin?
There are a few ways to ingest data
If you have minion then try out the Insert Into command as well
Segments should all in one format
s
Copy code
aws s3 cp data.json s3://path/to/data/tmp/
in query console:
Copy code
SET taskName = 'test';
SET input.fs.className = 'org.apache.pinot.plugin.filesystem.S3PinotFS';
INSERT INTO "test9"
FROM FILE 's3://path/to/data/tmp/'
failed:
Copy code
Error Code: 200
QueryExecutionError:
org.apache.hc.core5.http.HttpException: Unable to get tasks states map. Error code 415, Error message: {"code":415,"error":"HTTP 415 Unsupported Media Type"}
	at org.apache.pinot.common.minion.MinionClient.executeTask(MinionClient.java:130)
	at org.apache.pinot.core.query.executor.sql.SqlQueryExecutor.executeDMLStatement(SqlQueryExecutor.java:100)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:191)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:125)
The pod the minion is running on has aws s3 permissions from its service account.
I put ingestionConfig for the table like:
Copy code
"ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY",
      "batchConfigMaps": [
        {
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "us-west-2",
          "inputDirURI": "s3://path/to/data/tmp/",
          "includeFileNamePattern": "glob:**/*.json",
          "excludeFileNamePattern": "glob:**/*.tmp",
          "inputFormat": "json"
        }
      ]
    }
  },
  "task": {
    "taskTypeConfigsMap": {
      "SegmentGenerationAndPushTask": {
        "schedule": "0 * * * * ?",
        "tableMaxNumTasks": "10"
      }
    }
  }
and see it's able to list the files from s3:
Copy code
Execute CronJob: table - test9_OFFLINE, task - SegmentGenerationAndPushTask at 2025-05-16T00:40:00.001+0000
Trying to schedule task type: SegmentGenerationAndPushTask, isLeader: false
Trying to extract input files from path: s3://path/to/data/tmp/, and exclude input files from existing segments metadata: [s3://path/to/data/tmp/data.json], and input files from running tasks: []
Listed 1 files from URI: s3://path/to/data/tmp/, is recursive: true
Final input files for task config generation: []
Finished CronJob: table - test9_OFFLINE, task - SegmentGenerationAndPushTask, next runtime is 2025-05-16T00:41:00.000+000
image.png
x
OK, seems to be auth issue still. I will check
s
I've tried few more things without success...
Copy code
./bin/pinot-admin.sh CreateSegment -> UploadSegment
./bin/pinot-admin.sh GenerateData -> ImportData
Download segment from s3 -> curl -X POST /segments
Download segment from s3 -> curl -X POST /v2/segments
The segments get uploaded and I see them on s3 fine, but the segment status is BAD
Copy code
java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.lang.UnsupportedOperationException
	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchUntarSegmentToLocalStreamed(BaseSegmentFetcher.java:109) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndStreamUntarToLocal(SegmentFetcherFactory.java:147) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndStreamUntarToLocal(SegmentFetcherFactory.java:154) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:786) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:746) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndLoadSegment(BaseTableDataManager.java:389) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.core.data.manager.BaseTableDataManager.addNewOnlineSegment(BaseTableDataManager.java:360) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.doAddOnlineSegment(OfflineTableDataManager.java:54) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.core.data.manager.BaseTableDataManager.addOnlineSegment(BaseTableDataManager.java:313) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOnlineSegment(HelixInstanceDataManager.java:275) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:131) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1b9310a480789da998a2243c87d8e395fc6ba69a]
	... 12 more
Is the segment itself corrupted?
x
Can you try to edit the basic auth and check if the data flows?
And we can review the auth config for the components