Hi Team, I am trying to perform a filesystem ingestion to gcs bucket using minion SegmentGenerationAndPushTask. The example in the docs (https://docs.pinot.apache.org/basics/concepts/components/cluster/minion#segmentgenerationandpushtask) describes the config when fetching files form s3:

"ingestionConfig": {
    "batchIngestionConfig": {
      "batchConfigMaps": [
        {
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "us-west-2",
          "input.fs.prop.secretKey": "....",
          "input.fs.prop.accessKey": "....",
          "inputDirURI": "<s3://my.s3.bucket/batch/airlineStats/rawdata/>",
          ...

we have updated className to:

org.apache.pinot.plugin.filesystem.GcsPinotFS

, but we cannot fiure how to set the

gcpKey

instead of

secretKey

and

accessKey

properties. Probably we need to set gcp

projectId

as well.

you can use the gcp key file by copying it to pinot local and point to the file location

which property should I use to point to the key file? eg something like

input.fs.prop.gcpKey

?

You can do something like:

"input.fs.className": "org.apache.pinot.plugin.filesystem.GcsPinotFS",
          "inputDirURI": "<gs://my-pinot-data/>",
          "input.fs.prop.projectId": "my-project-id",
          "input.fs.prop.region": "asia-southeast1",
          "input.fs.prop.jsonKey": "{\"type\": \"service_account\",\n  \"project_id\": \"my-project-id\",\n  \"private_key_id\": \"398f440efd0eb122324478581723\",\n  \"private_key\": \"-----BEGIN PRIVATE KEY-----\naaaaaaaaaaaaaaa==\n-----END PRIVATE KEY-----\n\",\n  \"client_email\": \"<mailto:aaaa@my-project.iam.gserviceaccount.com|aaaa@my-project.iam.gserviceaccount.com>\",\n  \"client_id\": \"12312312\",\n  \"auth_uri\": \"<https://accounts.google.com/o/oauth2/auth>\",\n  \"token_uri\": \"<https://oauth2.googleapis.com/token>\",\n  \"auth_provider_x509_cert_url\": \"<https://www.googleapis.com/oauth2/v1/certs>\",\n  \"client_x509_cert_url\": \"<https://www.googleapis.com/robot/v1/metadata/x509/aaa%40my-project-id.iam.gserviceaccount.com>\",\n  \"universe_domain\": \"<http://googleapis.com|googleapis.com>\"}",
          "inputFormat": "parquet",
          "includeFileNamePattern": "glob:**/*.parquet",
          "excludeFileNamePattern": "glob:**/*.tmp"

mmm, we already have the file mouned in the pinot pods.. we tried this property with a path to the key, but it didn't work. We woud like to avoid writing the key like this ..

i see, then use

"input.fs.prop.gcpKey": "/my/gcp/key/path"