This message was deleted.

You could try using a time function. https://druid.apache.org/docs/latest/misc/math-expr.html#time-functions

getting this err when used in transformspec { "ingestionState": "BUILD_SEGMENTS", "unparseableEvents": {}, "rowStats": { "buildSegments": { "processed": 0, "processedWithError": 0, "thrownAway": 0, "unparseable": 0 } }, "errorMsg": "org.apache.druid.java.util.common.ISE: Could not transform value for kafka.timestamp reason: Function[timestamp] first argument should be a STRING but got LONG instead\n\tat org.apache.druid.segment.transform.ExpressionTransform$ExpressionRowFunction.eval(ExpressionTransform.java:117)\n\tat org.apache.druid.segment.transform.Transformer$TransformedInputRow.getRaw(Transformer.java:218)\n\tat org.apache.druid.segment.incremental.IncrementalIndex.toIncrementalIndexRow(IncrementalIndex.java:581)\n\tat org.apache.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:516)\n\tat org.apache.druid.segment.realtime.plumber.Sink.add(Sink.java:184)\n\tat org.apache.druid.segment.realtime.appenderator.StreamAppenderator.add(StreamAppenderator.java:276)\n\tat org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.append(BaseAppenderatorDriver.java:411)\n\tat org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.add(StreamAppenderatorDriver.java:189)\n\tat org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:654)\n\tat org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:266)\n\tat org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:151)\n\tat org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477)\n\tat org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:750)\n\tSuppressed: java.lang.RuntimeException: java.lang.IllegalArgumentException: fromIndex(0) > toIndex(-1)\n\t\tat org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:241)\n\t\tat org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:772)\n\t\t... 8 more\n\tCaused by: java.lang.IllegalArgumentException: fromIndex(0) > toIndex(-1)\n\t\tat java.util.ArrayList.subListRangeCheck(ArrayList.java:1016)\n\t\tat java.util.ArrayList.subList(ArrayList.java:1006)\n\t\tat org.apache.druid.segment.realtime.appenderator.StreamAppenderator.persistAll(StreamAppenderator.java:589)\n\t\tat org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:233)\n\t\t... 9 more\nCaused by: org.apache.druid.math.expr.ExpressionValidationException: Function[timestamp] first argument should be a STRING but got LONG instead\n\tat org.apache.druid.math.expr.NamedFunction.validationFailed(NamedFunction.java:46)\n\tat org.apache.druid.math.expr.Function$TimestampFromEpochFunc.apply(Function.java:2829)\n\tat org.apache.druid.math.expr.FunctionExpr.eval(FunctionalExpr.java:183)\n\tat org.apache.druid.segment.transform.ExpressionTransform$ExpressionRowFunction.eval(ExpressionTransform.java:113)\n\t... 16 more\n", "segmentAvailabilityConfirmed": false, "segmentAvailabilityWaitTimeMs": 0 }

I think it already comes out as long from Kafka?

it is coming as string, is there any possibility to do it instead of adding in dimension. since we are not aware of all dimensions so we kept our ingestion spec like this "dimensionsSpec": { "dimensions": [], "dimensionExclusions": [ "__time", "ts" ], "includeAllDimensions": true } if we define only kafka.timestamp in dimension other dimensions are not seen in the query

ah i see, this is the schemaless dimensions setup (https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#inclusions-and-exclusions)

Can we do it using transformSpec, so it will be simple? A final option that we see is using a query, just if data is already converted to the required format we don't need conversions in query :)

What version are you running?

we are using 24.0.0

Hrm... From reading the code, that includeAllDimensions should work. When you say you are only getting kafka.timestamp and nothing else, is that from the sampling in the wizard in the console or are you actually running the ingestion and not getting the columns?

this our sample data: {"ts":"2021-11-29T174500+08:00","mt":"pwr","p1":"BLR","p2":"767","p3":"767","p4":"171","p5":"244","p6":"08","p7":"001","p8":"001","p9":"test","p12345":930,"p233434":900,"duration":15} when we ingest this data with the above spec in the SQL console we see only "kafka.timestamp"(so basically it is showing only 'kafka.timestamp' and 'ts' and other columns are not shown in the SQL query tab)

Oh, do you have the supervisor/task still running?

yes

Can you pause the supervisor?

we paused(suspended) the supervisor and tried refreshing browser, but still, we see the same.

Can you go to the segments view, click on the magnifying glass to the right of the line for that one segment and copy&paste or screenshot the Payload section of that?

{ "dataSource": "test", "interval": "2021-11-29T090000.000Z/2021-11-29T100000.000Z", "version": "2022-10-31T052448.269Z", "loadSpec": { "type": "hdfs", "path": "hdfs://hadoop.local:8020/druid/segments/test/20211129T090000.000Z_20211129T100000.000Z/2022-10-31T05_24_48.269Z/0_1c5681a1-6443-4bad-81b9-718216fbfa7b_index.zip" }, "dimensions": "kafka.timestamp", "metrics": "", "shardSpec": { "type": "numbered", "partitionNum": 0, "partitions": 0 }, "binaryVersion": 9, "size": 739, "identifier": "test_2021-11-29T090000.000Z_2021-11-29T100000.000Z_2022-10-31T052448.269Z" }

That is the one... and hrm. That is weird. I'm not sure exactly what is going on, it's definitely not what I expect. If it's possible, you can use the nested column to do auto-discovered columns, which will also identify types as well

can u plz share some links or examples for this nested column?

will go through this , Thank you.

Hi Naresh, one way to do this is to pre-process the data. In other words, send the data from Kafka to a real time stream processor, convert the string to long, and deliver to Druid.