https://pinot.apache.org/ logo
Join Slack
Powered by
Will it convert to timestamp format or will it sti...
# general
h
Will it convert to timestamp format or will it still be epoch values ?
m
You specified data type as LONG, so it will be LONG
What is the behavior you would like?
h
like a datetime
m
You don’t really care about how it is stored right? You should want to be able to query using timestamp format?
h
it is not showing/recognizing as date time format in superset
and that is limiting the charts creation
m
Also cc: @Xiang Fu for more comments.
h
Sure. i tried couple of things but nothing worked
This is my sample data in kafka :{ "ordertime": 1505487555456, "orderid": 110300, "itemid": "Item_202", "orderunits": 7.3152506426088175, "address": { "city": "City_71", "state": "State_33", "zipcode": 58184 } }
m
What’s the query superset is issuing?
h
this is my schema in pinot table: "dateTimeFieldSpecs": [ { "name": "ordertime", "dataType": "LONG", "format": "1MILLISECONDSEPOCH", "granularity": "1:MILLISECONDS" } ]
I'm looking for any ways to represent order time as timestamp in pinot and it shows like 120000
x
Copy code
"dataType": "LONG", -> "TIMESTAMP"
Copy code
TIMESTAMP index can only be created on TIMESTAMP data type.
h
That worked!
👍 1
getting this error on superset side. anything you have seen before
image.png
x
Hmm No
Are you on newest pinotdb python lib?
h
yes
x
Hmm
Did you see any error from superset side
And you may need to edit the superset table column metadata
h
@Andrew Sunarto
a
Ah, okay, found the option to change column to temporal. Getting this error when trying to use Pinot table in Superset calendar heatmap: Unexpected Error Unknown string format
Copy code
Traceback (most recent call last):
  File "/usr/local/lib64/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 2059, in objects_to_datetime64ns
    values, tz_parsed = conversion.datetime_to_datetime64(data)
  File "pandas/_libs/tslibs/conversion.pyx", line 350, in pandas._libs.tslibs.conversion.datetime_to_datetime64
TypeError: Unrecognized value type: <class 'str'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/superset/viz.py", line 532, in get_df_payload
    df = self.get_df(query_obj)
  File "/usr/local/lib/python3.6/site-packages/superset/viz.py", line 321, in get_df
    df[DTTM_ALIAS], utc=False, format=timestamp_format
  File "/usr/local/lib64/python3.6/site-packages/pandas/core/tools/datetimes.py", line 803, in to_datetime
    values = convert_listlike(arg._values, format)
  File "/usr/local/lib64/python3.6/site-packages/pandas/core/tools/datetimes.py", line 466, in _convert_listlike_datetimes
    allow_object=True,
  File "/usr/local/lib64/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 2064, in objects_to_datetime64ns
    raise e
  File "/usr/local/lib64/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 2055, in objects_to_datetime64ns
    require_iso8601=require_iso8601,
  File "pandas/_libs/tslib.pyx", line 352, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 579, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 714, in pandas._libs.tslib.array_to_datetime_object
  File "pandas/_libs/tslib.pyx", line 705, in pandas._libs.tslib.array_to_datetime_object
  File "pandas/_libs/tslibs/parsing.pyx", line 243, in pandas._libs.tslibs.parsing.parse_datetime_string
  File "/usr/lib/python3.6/site-packages/dateutil/parser.py", line 1182, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python3.6/site-packages/dateutil/parser.py", line 559, in parse
    raise ValueError("Unknown string format")
ValueError: Unknown string format
ordertime column looks like this:
x
seems the issue of time parser
Copy code
File "pandas/_libs/tslibs/parsing.pyx", line 243, in pandas._libs.tslibs.parsing.parse_datetime_string
this graph cannot recognize the pinot date format?
a
seems like it. I’m trying to change some other Superset settings to be able to read the column correctly.
Open in Slack
PreviousNext
Powered by