I have a Kafka topic with AVRO encoding. The time ...
# general
n
I have a Kafka topic with AVRO encoding. The time column is of type (long). The logical type is (timestamp-micros). Any way to convert it to milliseconds and defined a datetime field spec like the following (without having to create a new column):
Copy code
{
  "name": "event_time_ms",
  "dataType": "TIMESTAMP",
  "format": "1:MILLISECONDS:TIMESTAMP",
  "granularity": "1:MILLISECONDS"
}
@Jackie
m
Hey @Nizar Hejazi I expect we should be able to perform the transform in-place. Let me find out.
👍 1
t
Hey @Nizar Hejazi, currently you would have to create another derived column otherwise you would end up with a cyclic transform error.
The transform function would look something like this
Copy code
"transformConfigs": [
  {
    "columnName": "event_time_ms",
    "transformFunction": "\"event_time_micro\" / 1000"
  }
],
n
Thanks @Tim Santos. Do we know why:
Copy code
timeConvert(col,'MICROSECONDS','MILLISECONDS')
does not work (timeConvert cannot be recognized)?
j
timeConvert
is not a supported scalar function.
dateTrunc
is supported, but it is more expensive than directly do the
div
👍 1