hi anyone can show me example config realtime table with kafka sasl_ssl.

https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#some-more-kafka-ingestion-configs Kafka consumer configs are transparently passed

this is my table code { "tableName": "user_stream_kkts_test", "tableType": "REALTIME", "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "segmentsConfig": { "schemaName": "user_stream_kkts", "timeColumnName": "timestampInEpoch", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig": null, "crypterClassName": null, "peerSegmentDownloadScheme": null }, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [], "sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null, "aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "user_stream", "stream.kafka.broker.list": "kkts019094,kkts029094,kkts03:9094", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M", "stream.kafka.consumer.prop.group.id": "rdp_lookup", "security.protocol": "SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name": "bigfoot", "ssl.truststore.location": "/data/apache-pinot/keytab/kfuat_rdp/kkts.client.truststore.jks", "ssl.truststore.password": "godzilla007", "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab principal=kfuat_rdp@KAFKA.SECURE;" } }, "metadata": {}, "ingestionConfig": { "filterConfig": null, "transformConfigs": null }, "quota": { "storage": null, "maxQueriesPerSecond": null }, "task": null, "routing": { "segmentPrunerTypes": null, "instanceSelectorType": null }, "query": { "timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null, "tierConfigs": null }

does this config work with Kafka console consumer ?

client.property file is

are you using k8s or baremetal

i use baremetal. what is parameter for specify keytab file @Xiang Fu

so something like

sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required useKeyTab=true storeKey=true keyTab="/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab" principal="kfuat_rdp@KAFKA.SECURE";

it's Error: Bad Request

have you tried to escape some characters?

i try this { "tableName": "user_stream_kkts", "tableType": "REALTIME", "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "segmentsConfig": { "schemaName": "user_stream_kkts", "timeColumnName": "timestampInEpoch", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig": null, "crypterClassName": null, "peerSegmentDownloadScheme": null }, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [], "sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null, "aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "user_stream", "stream.kafka.broker.list": "kkts019094,kkts029094,kkts03:9094", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M", "stream.kafka.consumer.group.id": "rdp_lookup", "security.protocol": "SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name": "bigfoot", "ssl.truststore.location": "/data/apache-pinot/keytab/kfuat_rdp/kkts.client.truststore.jks", "ssl.truststore.password": "godzilla007", "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab\" principal=\"kfuat_rdp@KAFKA.SECURE\";" } }, "metadata": {}, "ingestionConfig": { "filterConfig": null, "transformConfigs": [] }, "quota": { "storage": null, "maxQueriesPerSecond": null }, "task": null, "routing": { "segmentPrunerTypes": null, "instanceSelectorType": null }, "query": { "timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null, "tierConfigs": null }

i think this config is per pinot table based?

i try to config per pinot table but server still use /etc/krb5.conf on deafault. did you have parameter for krb5.conf config on table ? @Xiang Fu

Hmm, so you have many krb files?

yes we have many kdc server.

It should be through the jaas config for Krb5LoginModule

you mean it shoud config on this --> "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=\"/data/apache-pinot/keytab/U-SVC-RDP.keytab\" principal=\"U-SVC-RDP@TRUE.TH\" doNotPrompt=false;"

Yes, it's similar to Kafka consumer client config

ok i'll try to config

hmm

JSONPATHSTRING(order,'$.channel') it corrent but can not create table.

hmm, do you have more logs from the controller? this seems to be the evaluation error

can you recommend me about change column, when i test change ward in jason from order to hello it's wotking. but on real data it order...

can you add

order

into schema, something like:

{
  "schemaName": "omx_order_20",
  "dimensionFieldSpecs": [
    {
      "name": "channel",
      "dataType": "STRING"
    },
    {
      "name": "orderId",
      "dataType": "STRING"
    },
    {
      "name": "customer",
      "dataType": "STRING"
    },
    {
      "name": "omxtrackingID",
      "dataType": "STRING"
    },
    {
      "name": "order",
      "dataType": "JSON"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "submissionDate",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HH:mm:ss.SSSZ",
      "granularity": "1:MILLISECONDS"
    }
  ]
}