https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
b

Bordin Suwannatri

03/15/2022, 3:34 PM
hi anyone can show me example config realtime table with kafka sasl_ssl.
b

Bordin Suwannatri

03/15/2022, 7:19 PM
this is my table code { "tableName": "user_stream_kkts_test", "tableType": "REALTIME", "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "segmentsConfig": { "schemaName": "user_stream_kkts", "timeColumnName": "timestampInEpoch", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig": null, "crypterClassName": null, "peerSegmentDownloadScheme": null }, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [], "sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null, "aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "user_stream", "stream.kafka.broker.list": "kkts019094,kkts029094,kkts03:9094", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M", "stream.kafka.consumer.prop.group.id": "rdp_lookup", "security.protocol": "SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name": "bigfoot", "ssl.truststore.location": "/data/apache-pinot/keytab/kfuat_rdp/kkts.client.truststore.jks", "ssl.truststore.password": "godzilla007", "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab principal=kfuat_rdp@KAFKA.SECURE;" } }, "metadata": {}, "ingestionConfig": { "filterConfig": null, "transformConfigs": null }, "quota": { "storage": null, "maxQueriesPerSecond": null }, "task": null, "routing": { "segmentPrunerTypes": null, "instanceSelectorType": null }, "query": { "timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null, "tierConfigs": null }
Copy code
{
  "_code": 500,
  "_error": "org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"
}
@Xiang Fu please help
x

Xiang Fu

03/15/2022, 7:34 PM
does this config work with Kafka console consumer ?
If you can consume with Kafka console consumer config file, then you should be able to use same configs to ingest from Pinot
b

Bordin Suwannatri

03/16/2022, 4:52 AM
client.property file is
security.protocol=SASL_SSL sasl.mechanism=GSSAPI sasl.kerberos.service.name=bigfoot ssl.truststore.location=kkts.client.truststore.jks ssl.truststore.password=godzilla007 sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;
client jaas file.
i test comsume with client property it work fine.
how can i put parameter jaas and keytab and etc.
how to see document of all parameter for streamConfigs
Mar 16 121442 poc-pinot01 start-controller.sh: Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
look like pinot not found my keytab file @Xiang Fu
x

Xiang Fu

03/16/2022, 6:32 AM
are you using k8s or baremetal
can u specify the location of keytab file?
b

Bordin Suwannatri

03/16/2022, 8:33 AM
i use baremetal. what is parameter for specify keytab file @Xiang Fu
i use this patameter "sasl.jaas.config": "/data/apache-pinot/keytab/kfuat_rdp/client_jaas.conf" but it not working . error is ---> Mar 16 120853 poc-pinot01 start-controller.sh: Caused by: java.lang.IllegalArgumentException: Value not specified for key 'keyTab' in JAAS config
i try to change it to username/ password "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=(kfuat_rdp) password=(hello);" it's still not wotking. error is -->Mar 16 124728 poc-pinot01 start-controller.sh: Caused by: java.lang.IllegalArgumentException: Value not specified for key 'username' in JAAS config
my localtion of keytab file is jaas file. (/data/apache-pinot/keytab/kfuat_rdp/client_jaas.conf) cat /data/apache-pinot/keytab/kfuat_rdp/client_jaas.conf Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab" principal="kfuat_rdp@KAFKA.SECURE"; };
x

Xiang Fu

03/16/2022, 8:54 AM
so something like
Copy code
sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required useKeyTab=true storeKey=true keyTab="/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab" principal="kfuat_rdp@KAFKA.SECURE";
b

Bordin Suwannatri

03/16/2022, 8:57 AM
it's Error: Bad Request
x

Xiang Fu

03/16/2022, 9:46 AM
have you tried to escape some characters?
or this one:
Copy code
sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab" principal="kfuat_rdp@KAFKA.SECURE";
b

Bordin Suwannatri

03/16/2022, 9:47 AM
i try this { "tableName": "user_stream_kkts", "tableType": "REALTIME", "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "segmentsConfig": { "schemaName": "user_stream_kkts", "timeColumnName": "timestampInEpoch", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig": null, "crypterClassName": null, "peerSegmentDownloadScheme": null }, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [], "sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null, "aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "user_stream", "stream.kafka.broker.list": "kkts019094,kkts029094,kkts03:9094", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M", "stream.kafka.consumer.group.id": "rdp_lookup", "security.protocol": "SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name": "bigfoot", "ssl.truststore.location": "/data/apache-pinot/keytab/kfuat_rdp/kkts.client.truststore.jks", "ssl.truststore.password": "godzilla007", "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab\" principal=\"kfuat_rdp@KAFKA.SECURE\";" } }, "metadata": {}, "ingestionConfig": { "filterConfig": null, "transformConfigs": [] }, "quota": { "storage": null, "maxQueriesPerSecond": null }, "task": null, "routing": { "segmentPrunerTypes": null, "instanceSelectorType": null }, "query": { "timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null, "tierConfigs": null }
i walrey try this still error.
Copy code
sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab" principal="kfuat_rdp@KAFKA.SECURE";
@Xiang Fu we use this it working "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=\"/data/apache-pinot/keytab/U-SVC-RDP.keytab\" principal=\"U-SVC-RDP@TRUE.TH\" doNotPrompt=false;"
now if i have multiple KDC how can i config on table. example table 1 realm kdc1 table2 realm kdc2 i try to do that separate krb5.conf. it's not work. did you have some parameter or example ? please share to me. @Mayank @Xiang Fu
x

Xiang Fu

03/21/2022, 6:18 PM
i think this config is per pinot table based?
b

Bordin Suwannatri

03/22/2022, 6:31 AM
i try to config per pinot table but server still use /etc/krb5.conf on deafault. did you have parameter for krb5.conf config on table ? @Xiang Fu
x

Xiang Fu

03/22/2022, 6:59 AM
Hmm, so you have many krb files?
b

Bordin Suwannatri

03/22/2022, 6:59 AM
yes we have many kdc server.
many kafka cluster use separate keberlos server.
x

Xiang Fu

03/22/2022, 7:00 AM
It should be through the jaas config for Krb5LoginModule
In that you can specify the krb file location
b

Bordin Suwannatri

03/22/2022, 7:00 AM
you mean it shoud config on this --> "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=\"/data/apache-pinot/keytab/U-SVC-RDP.keytab\" principal=\"U-SVC-RDP@TRUE.TH\" doNotPrompt=false;"
ABout table realtime config datetime column is require ?
x

Xiang Fu

03/22/2022, 8:16 AM
Yes, it's similar to Kafka consumer client config
You can config it per table basis or it will use the default one
b

Bordin Suwannatri

03/22/2022, 8:22 AM
ok i'll try to config
@Xiang Fu why i can not use jsonPathString with order ? "transformFunction": "jsonPathString(order,'$.channel')"
if i use another word like ordera , orderb it can create table .
but on my json data is format is --> { "order": {"channel": "SFF","orderId": "22031500DRS020020016"}, "customer": "570809", "omxtrackingID": "99-d173c048-8bf2-4261-a440-36d1045c63e2", "submissionDate": "2022-03-15T173144.540+0700" }
can not use "order" in jsonPathString
x

Xiang Fu

03/22/2022, 10:57 AM
hmm
any error getting from pinot?
is
order
,
channel
a column in your schema
"transformFunction": "JSONPATHSTRING(order,'$.channel')"
is correct .
b

Bordin Suwannatri

03/22/2022, 3:48 PM
JSONPATHSTRING(order,'$.channel') it corrent but can not create table.
my schema { "schemaName": "omx_order_20", "dimensionFieldSpecs": [ { "name": "channel", "dataType": "STRING" }, { "name": "orderId", "dataType": "STRING" } ], "dateTimeFieldSpecs": [ { "name": "submissionDate", "dataType": "LONG", "format": "1MILLISECONDSEPOCH", "granularity": "1:MILLISECONDS" } ] }
the real schema test that iuse is { "schemaName": "omx_order_20", "dimensionFieldSpecs": [ { "name": "channel", "dataType": "STRING" }, { "name": "orderId", "dataType": "STRING" }, { "name": "customer", "dataType": "STRING" }, { "name": "omxtrackingID", "dataType": "STRING" } ], "dateTimeFieldSpecs": [ { "name": "submissionDate", "dataType": "TIMESTAMP", "format": "1MILLISECONDSSIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HHmmss.SSSZ", "granularity": "1:MILLISECONDS" } ] }
i use this is work. but on my data not person it's "order" "transformConfigs": [ { "columnName": "channel", "transformFunction": "jsonPathString(person,'$.channel')" }, { "columnName": "orderId", "transformFunction": "jsonPathString(person,'$.orderId')" } ]
Copy code
{
  "_code": 400,
  "_error": "Invalid transform function 'jsonPathString(order,'$.channel')' for column 'channel'"
}
x

Xiang Fu

03/22/2022, 6:20 PM
hmm, do you have more logs from the controller? this seems to be the evaluation error
I think you need to config column order as well
can you try to create a new column order
b

Bordin Suwannatri

03/23/2022, 3:29 AM
can you recommend me about change column, when i test change ward in jason from order to hello it's wotking. but on real data it order...
about log how to write log to file. now i use systemctl start service pinot-controller it write log on /var/log/massage.
x

Xiang Fu

03/23/2022, 7:44 PM
can you add
order
into schema, something like:
Copy code
{
  "schemaName": "omx_order_20",
  "dimensionFieldSpecs": [
    {
      "name": "channel",
      "dataType": "STRING"
    },
    {
      "name": "orderId",
      "dataType": "STRING"
    },
    {
      "name": "customer",
      "dataType": "STRING"
    },
    {
      "name": "omxtrackingID",
      "dataType": "STRING"
    },
    {
      "name": "order",
      "dataType": "JSON"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "submissionDate",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HH:mm:ss.SSSZ",
      "granularity": "1:MILLISECONDS"
    }
  ]
}