Hi again, folks! Hey, I got a question about timestamps in datetime columns: I'm trying to use

1:MILLISECONDS:EPOCH

, and I'm publishing Kafka events containing timestamps that are basically

int(time_in_seconds_as_float * 1000)

from a Python-based app, but when I use the incubator to query the table I'm getting back negative values. I'm probably doing something wrong, but isn't the idea to publish the time, in milliseconds, since Epoch (1970-01-01 000000)?

How is

time_in_seconds_as_float

generated? I am guessing it is not in EPOCH?

Hi @Mayank! Here it is: https://docs.python.org/3/library/time.html#time.time - I'm using

datetime.timestamp()

, but it follows the same specifications, so it's the time since Epoch in seconds, but as a float containing digits for microsecond resolution.

Could you share the table config/schema?

Sure! I'll share only the relevant parts, but let me know if you need the whole of it. Here's the part of the schema:

"dateTimeFieldSpecs": [
    {
      "name": "timestamp",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]

and here's the relevant part of the table config:

"segmentsConfig": {
      "timeColumnName": "timestamp",
      "timeType": "MILLISECONDS",
      "replicasPerPartition": "1",
      "schemaName": "bb8_analyses_logs_schema"
    },

Hmm, not sure why this might be happening. May be there's float overflow, or precision loss before the value hits pinot?

Not really, no... in Python 3,

int

is actually implemented as long, so there's no problem with that. I'll do some experiments and try tinkering with it. Is it possible to define seconds as float in Pinot? Like using

dataType

as

FLOAT

, then defining the format as

1:SECONDS:EPOCH

?

No sweat, glad you figured it out.