hello everyone i found some error whith transformFunction jsonPathString i can not use word order in jsonPathString --> "transformFunction": "jsonPathString(order,'$.channel')" -->this is not work. i test modify json replace from order to hello and user this --> "transformFunction": "jsonPathString(hello,'$.channel')" it's working. why i can not use "order". my real json massage they use "order". Please help.

without looking into it any further yet, it might be b/c 'order' is a reserved word in sql and this is likely going through a sql parser...but maybe you can quote the word order and see if that works:

"transformFunction": "jsonPathString(\"order\",'$.channel')"

oh it's work thank very much @User

you'll need to use a transformation function for that as well. You can see an example here - https://dev.startree.ai/docs/pinot/recipes/datetime-string-to-timestamp#add-schema-and-table

thank you.

you'll have to use a different column name than what is in the source data btw

on my json data type is string. "submissionDate" : "2022-03-15T173144.540+0700" I test to create table . it can create but not see data. schema structure that i test is. { "schemaName": "omx_order_20", "dimensionFieldSpecs": [ { "name": "channel", "dataType": "STRING" }, { "name": "orderId", "dataType": "STRING" } ], "dateTimeFieldSpecs": [ { "name": "submissionDate", "dataType": "STRING", "format": "1MILLISECONDSSIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HHmmss.SSSZ", "granularity": "1:MILLISECONDS" } ] }

oh as a string it should work. But if it's a string I don't think you'd be able to do proper data operations (in case you wanted to do that)

normally for real time table. we use dataFormat from system or in side json massage ?

unless you have a datetime column in your schema/table there won't be one, so you do need to specify the data yourself (e.g. via JSON message)

i try to create table. it can create but not found any Data. { "tableName": "omx_order_20", "tableType": "REALTIME", "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "segmentsConfig": { "schemaName": "omx_order_20", "timeColumnName": "submissionDate", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig": null, "crypterClassName": null, "peerSegmentDownloadScheme": null }, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [], "sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null, "aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "omx_order20", "stream.kafka.broker.list": "172.19.131.55:9092", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M" } }, "metadata": {}, "ingestionConfig": { "filterConfig": null, "transformConfigs": [ { "columnName": "channel", "transformFunction": "jsonPathString(\"order\",'$.channel')" }, { "columnName": "orderId", "transformFunction": "jsonPathString(\"order\",'$.orderId')" }, { "columnName": "submissionDate", "transformFunction": "FromDateTime('$.submissionDate','YYYY-MM-dd''T''HHmmss.SSSZ')" } ] }, "quota": { "storage": null, "maxQueriesPerSecond": null }, "task": null, "routing": { "segmentPrunerTypes": null, "instanceSelectorType": null }, "query": { "timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null, "tierConfigs": null }

You can't use the JSON path syntax inside

FromDataTime

- it doesn't know what it means

oh

also I think you can only run one function at a time

i just want to create table consume kafka. i test with EPOCH on json. it no need to do anything. but for this --> "submissionDate" : "2022-03-15T173144.540+0700" we need to converse ?

epoch you don't need any time conversion

how can i do it bro.

ok .modify column name another name .

yeh - if you use a transform fn it can't transform a value to the same name

oh it's work.

you only need to create the columns that you need - don't need to create columns for every JSON property

Thank you bro. you help me so much. tomorrow i will select some column for test poc with trino.

logs are under

logs/pinot-all.log

on each component

thank you bro.

it stores it under

data.dir

which will be a tmp dir unless you explicitly set it

how can i use separate kerberlos on realtime table config? did you have some example for config separate kdc? this is my stream config. now i can connect only 1 kdc it default on /etc/krb5.conf we have multiple kafka and kdc need to connect... "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "PROD-json-MB-Postpaid-Sales-Online", "stream.kafka.broker.list": "tykbpr019092,tykbpr029092,tykbpr01:9092", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.segment.size": "100m", "stream.kafka.consumer.group.id": "rdp_lookup", "security.protocol": "SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name": "kafka", "ssl.truststore.location": "/data/apache-pinot/keytab/tykbpr.client.truststore.jks", "ssl.truststore.password": "xxxx", "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=\"/data/apache-pinot/keytab/U-SVC-RDP.keytab\" principal=\"U-SVC-RDP@TRUE.TH\" doNotPrompt=false;" } @User

https://docs.pinot.apache.org/basics/components/deep-store take a look at this and then the S3 example as well

thank you.