https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • d
    Or should I change my Pinot schema?
  • x
    there are two moe configs to infer timestamp and date type
  • x
    Copy code
    pinot.infer-date-type-in-schema=true
pinot.infer-timestamp-type-in-schema=true
  • x
    do you see the column type is timestamp when you do describe events_testing
  • x
    That image should have this change already
  • x
    this works for me now
  • x
    Copy code
    presto:default> select DATE_TRUNC('day', "dayssinceepoch") as mydate from airlineStats limit 10;
Query 20200501_003454_00012_y857r failed: line 1:8: Unexpected parameters (varchar(3), integer) for function date_trunc. Expected: date_trunc(varchar(x), date) , date_trunc(varchar(x), time) , date_trunc(varchar(x), time with time zone) , date_trunc(varchar(x), timestamp) , date_trunc(varchar(x), timestamp with time zone)
select DATE_TRUNC('day', "dayssinceepoch") as mydate from airlineStats limit 10

presto:default> select DATE_TRUNC('day', "dayssinceepoch") as mydate from airlineStats limit 10;
   mydate
------------
 2014-04-01
 2014-04-01
 2014-04-01
 2014-04-02
 2014-04-02
 2014-04-02
 2014-04-02
 2014-04-02
 2014-04-02
 2014-04-03
(10 rows)

Query 20200501_022002_00000_9icjz, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:02 [10 rows, 40B] [5 rows/s, 23B/s]
  • x
    Copy code
    presto:default> describe airlinestats;
        Column        |  Type   | Extra |  Comment
----------------------+---------+-------+-----------
 flightnum            | integer |       | DIMENSION
 origin               | varchar |       | DIMENSION
 quarter              | integer |       | DIMENSION
 lateaircraftdelay    | integer |       | DIMENSION
 divactualelapsedtime | integer |       | DIMENSION
 divwheelsons         | varchar |       | DIMENSION
 divwheelsoffs        | varchar |       | DIMENSION
 airtime              | integer |       | DIMENSION
 arrdel15             | integer |       | DIMENSION
 divtotalgtimes       | varchar |       | DIMENSION
 deptimeblk           | varchar |       | DIMENSION
 destcitymarketid     | integer |       | DIMENSION
 divairportseqids     | varchar |       | DIMENSION
 dayssinceepoch       | date    |       | TIME
 deptime              | integer |       | DIMENSION
 month                | integer |       | DIMENSION
.....
  • d
    @Xiang Fu - that error is during startup of Presto. It kills the server.
    Copy code
    2020-04-30T19:06:21.526-0700	ERROR	main	com.facebook.presto.server.PrestoServer	com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Configuration property 'pinot.infer-date-type-in-schema' was not used
  at com.facebook.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:238)

2) Configuration property 'pinot.infer-timestamp-type-in-schema' was not used
  at com.facebook.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:238)

2 errors
java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Configuration property 'pinot.infer-date-type-in-schema' was not used
  at com.facebook.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:238)

2) Configuration property 'pinot.infer-timestamp-type-in-schema' was not used
  at com.facebook.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:238)

2 errors
	at com.facebook.presto.pinot.PinotConnectorFactory.create(PinotConnectorFactory.java:94)
	at com.facebook.presto.connector.ConnectorManager.createConnector(ConnectorManager.java:364)
	at com.facebook.presto.connector.ConnectorManager.addCatalogConnector(ConnectorManager.java:222)
	at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:214)
	at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:200)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:123)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:98)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:80)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:68)
	at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:135)
	at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:77)
  • x
    hmmm
  • x
    let me check that image
  • x
    can you try image 
    0.234.3
    ?
  • d
    Ah, I don't see an easy download link for that version. I'll try it tomorrow morning.
  • d
    I'm trying to get the Presto docker container working again. I can't get Presto to finish starting up. The last line printed is one of the options. My laptop fan goes crazy when I run it. Any ideas?
    Copy code
    expired
2020-05-01T05:51:54.338Z	INFO	main	Bootstrap	transaction.max-finishing-concurrency                                        1                                                                          1                                                                          Maximum parallelism for committing or aborting a transaction
    I've tried a few different run setups and can't get any of them to work. E.g. the docker run command in the gitbook. I merged the etc in the docker directory with my own etc.
    Copy code
    docker run \                  
--network kafka_default \
--name=presto-coordinator \ 
-v "$(pwd)"/etc:/home/presto/etc \
-p 8080:8080 \
-d apachepinot/pinot-presto:0.234.3
  • x
    I think this is the one
  • x
    how many catalogs you have in etc
  • x
    Copy code
    2020-05-01T06:20:58.656Z	INFO	main	Bootstrap	transaction.max-finishing-concurrency                                        1                                                                          1                                                                          Maximum parallelism for committing or aborting a transaction
2020-05-01T06:21:04.353Z	WARN	main	com.facebook.airlift.jmx.JmxAgent	Cannot determine if JMX agent is already running (not an Oracle JVM?). Will try to start it manually.
2020-05-01T06:21:04.452Z	INFO	main	com.facebook.airlift.jmx.JmxAgent	JMX agent started and listening on 7b634ba97f04:42235
2020-05-01T06:21:11.644Z	WARN	node-state-poller-0	com.facebook.presto.metadata.HttpRemoteNodeState	Node state update request to <http://172.19.0.4:8080/v1/info/state> has not returned in 290503.50s
2020-05-01T06:21:13.851Z	WARN	query-management-2	com.facebook.presto.memory.RemoteNodeMemory	Memory info update request to <http://172.19.0.4:8080/v1/memory> has not returned in 290505.71s
    It took a few seconds for me to pass that logs also
  • d
    Hello everybody, I am creating a custom aggregator, at the moment i am trying to optimize it before deploy. I have a question about segments. I read that segments are like partitions of the table, thay are processed in parallel by Pinot. I have a question about sorting, i know that talking about sorting in a parallel execution is pretty silly, the execution of each segment will take different time so it is not possible to understand in advance the order, ok, but.. if i end my query with 
    ORDER BY id ASC
    (where id is an Integer). Are the documents inside each partition/segment sorted before the aggregation (lets suppose before processing the MAX() aggregator)? or the sorting will be done at the end? I am asking that because, obviously, the order matter in my case...if each segment is ordered i can optimize my aggregator much more. I know that i can use a timestamp like recommended by @Kishore G to understand the "order" of each document but if the documents coming in an "random" order i must deal with that and this means...more code, more checks = slower.
  • k
    @Damiano sorting happens at the very end
  • d
    ok so there are no possibilities, we must to deal with random order in segments during the aggregation process
  • d
    @Kishore G i finished the code it works but i am optimizing it.. how big are segments? is there a "fixed" length ?
  • k
    there is no fixed length and you should not depend on that
  • d
    because in my case more "spare" the docs came in and more points i have to manage
  • d
    yes i am asking to understand how big they are to test my function with a pseudo-real length, however ok...
  • k
    within each segment, the order is based on the input data
  • d
  • k
    so you can sort it on stockId, time
  • d
    i can insert ordered documents
  • d
    during the insert right?
  • k
    yes
1...929394...166Latest