Kewei Shang
06/16/2021, 2:13 PMManual cluster setup
(link)’s Using launcher scripts
section, I ran
export JAVA_OPTS="-Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-controller.log"
bin/pinot-admin.sh StartController \
-zkAddress localhost:2191 \
-controllerPort 9000
to start the controller.
However, the controller logs lots of warns like the following (in the thread), and <http://localhost:9000/>
returns a blank web UI. May I have some help please? Thanks. The docker version works for me but I want to install Pinot on our EC2 nodes for further PoC.Mark Needham
regions
dim table and cases
normal table. And then I run this query:
select areaName, lookUp('regions', 'Region', 'LTLAName', areaName)
from cases limit 10
But the error message says it doesn't find the lookup function:
[
{
"errorCode": 200,
"message": "QueryExecutionError:\norg.apache.pinot.core.query.exception.BadQueryRequestException: Unsupported function: lookup with 4 parameters\n\tat org.apache.pinot.core.operator.transform.function.TransformFunctionFactory.get(TransformFunctionFactory.java:189)\n\tat org.apache.pinot.core.operator.transform.TransformOperator.<init>(TransformOperator.java:56)\n\tat org.apache.pinot.core.plan.TransformPlanNode.run(TransformPlanNode.java:52)\n\tat org.apache.pinot.core.plan.SelectionPlanNode.run(SelectionPlanNode.java:83)\n\tat org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:94)\n\tat org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:33)\n\tat org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:234)\n\tat org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:155)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:139)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)"
}
]
Any ideas?Kewei Shang
06/16/2021, 8:01 PMselect DATETIMECONVERT(MIN(created), '1:MILLISECONDS:EPOCH', '1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd HH:mm:ss', '1:SECONDS') as min_created from delivery_order limit 1
failed with the following error (in slack thread). The created
column is of type:
{
"name": "created",
"dataType": "LONG",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}
Interestingly, the query select DATETIMECONVERT(created, '1:MILLISECONDS:EPOCH', '1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd HH:mm:ss', '1:SECONDS') as min_created from delivery_order limit 1
without MIN()
works fine. May I have some advice? Thanks.Sidd
06/16/2021, 9:12 PMJai Patel
06/16/2021, 10:11 PMKen Krugler
06/16/2021, 10:40 PMsegmentIngestionFrequency
, then are APPEND
and REFRESH
values for segmentIngestionType
essentially equivalent?RK
06/17/2021, 10:12 AMMark Needham
version: '3.7'
services:
pinot:
image: apachepinot/pinot:0.7.1
command: "QuickStart -type batch"
container_name: "pinot-quickstart"
volumes:
- ./data:/data
ports:
- "9000:9000"
- "8000:8000"
from pinotdb import connect
conn = connect(host='localhost', port=9000, path='/query/sql', scheme='http')
curs = conn.cursor()
curs.execute("""
SELECT * from cases LIMIT 10
""")
for row in curs:
print(row)
I get this error when I run the query:
Traceback (most recent call last):
File "query.py", line 5, in <module>
curs.execute("""
File "/home/markhneedham/.local/share/virtualenvs/pinot-playground-V0PLiJ36/lib/python3.8/site-packages/pinotdb/db.py", line 44, in g
return f(self, *args, **kwargs)
File "/home/markhneedham/.local/share/virtualenvs/pinot-playground-V0PLiJ36/lib/python3.8/site-packages/pinotdb/db.py", line 289, in execute
self.check_sufficient_responded(
File "/home/markhneedham/.local/share/virtualenvs/pinot-playground-V0PLiJ36/lib/python3.8/site-packages/pinotdb/db.py", line 253, in check_sufficient_responded
raise exceptions.DatabaseError(
pinotdb.exceptions.DatabaseError: Query
SELECT * from cases LIMIT 10
timed out: Out of -1, only -1 responded, while needed was -1
Am I querying on the right port? In the examples port 8009 is used, but I tried that and got a different error!Pedro Silva
06/17/2021, 5:13 PM[
{
"errorCode": 410,
"message": "BrokerResourceMissingError"
}
]
The broker logs show this exception:
2021/06/17 16:47:13.084 WARN [BaseInstanceSelector] [ClusterChangeHandlingThread] Failed to find servers hosting segment: HitExecutionView__10__19__20210616T1108Z for table: HitExecutionView_REALTIME (all ONLINE/CONSUMING instances: [] and OFFLINE instances: [] are disabled, counting segment as unavailable)
2021/06/17 17:05:18.183 ERROR [BrokerResourceOnlineOfflineStateModelFactory] [HelixTaskExecutor-message_handle_thread] Caught exception while processing transition from OFFLINE to ONLINE for table: hitexecutionview_REALTIME
java.lang.IllegalStateException: Failed to find ideal state for table: hitexecutionview_REALTIME
at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.broker.routing.RoutingManager.buildRouting(RoutingManager.java:309) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.broker.broker.helix.BrokerResourceOnlineOfflineStateModelFactory$BrokerResourceOnlineOfflineStateModel.onBecomeOnlineFromOffline(BrokerResourceOnlineOfflineStateModelFactory.java:80) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
2021/06/17 17:05:18.283 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Exception while executing a state transition task hitexecutionview_REALTIME
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.IllegalStateException: Failed to find ideal state for table: hitexecutionview_REALTIME
at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.broker.routing.RoutingManager.buildRouting(RoutingManager.java:309) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.broker.broker.helix.BrokerResourceOnlineOfflineStateModelFactory$BrokerResourceOnlineOfflineStateModel.onBecomeOnlineFromOffline(BrokerResourceOnlineOfflineStateModelFactory.java:80) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
... 12 more
2021/06/17 17:05:18.301 ERROR [StateModel] [HelixTaskExecutor-message_handle_thread] Default rollback method invoked on error. Error Code: ERROR
2021/06/17 17:05:18.383 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread] Message execution failed. msgId: 366af265-24b4-4b59-a28f-1e6387e5d2aa, errorMsg: java.lang.reflect.InvocationTargetException
2021/06/17 17:05:18.392 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Skip internal error. errCode: ERROR, errMsg: null
2021/06/17 17:06:50.697 WARN [RoutingManager] [HelixTaskExecutor-message_handle_thread] Routing does not exist for table: hitexecutionview_REALTIME, skipping refreshing segment
2021/06/17 17:06:51.179 WARN [RoutingManager] [HelixTaskExecutor-message_handle_thread] Routing does not exist for table: hitexecutionview_REALTIME, skipping refreshing segment
2021/06/17 17:07:22.699 WARN [RoutingManager] [HelixTaskExecutor-message_handle_thread] Routing does not exist for table: hitexecutionview_REALTIME, skipping refreshing segment
2021/06/17 17:07:23.997 WARN [RoutingManager] [HelixTaskExecutor-message_handle_thread] Routing does not exist for table: hitexecutionview_REALTIME, skipping refreshing segment
2021/06/17 17:07:25.800 WARN [RoutingManager] [HelixTaskExecutor-message_handle_thread] Routing does not exist for table: hitexecutionview_REALTIME, skipping refreshing segment
Any ideas?Sumit Gupta
06/18/2021, 12:34 AMCarl
06/18/2021, 11:38 AMNeil Teng
06/18/2021, 3:06 PMNeil Teng
06/18/2021, 9:36 PMCarl
06/19/2021, 3:17 AMAtri Sharma
06/19/2021, 6:26 AMSanthosh CT
06/21/2021, 6:14 AMNeil Teng
06/21/2021, 7:59 PMNeil Teng
06/22/2021, 3:26 PMSELECT approx_distinct(id) AS "count"
FROM table
WHERE start_date <= current_date() AND end_date >= current_date()
BTW, what about where cat in ('a', 'b', 'c')?Carl
06/22/2021, 4:21 PMJackie
06/22/2021, 5:50 PMEvan Galpin
06/22/2021, 8:32 PMLian Jiang
06/22/2021, 8:40 PMNeil Teng
06/22/2021, 10:06 PMZsolt Takacs
06/23/2021, 3:47 PMReal-Time Pinot table: In case of real-time tables, make sure the "_pinot.server.instance.reload.consumingSegment_" config is set to true inside Server config. Without this, the current consuming segment(s) will not reflect the default null value for newly added columns.
Zsolt Takacs
06/23/2021, 3:49 PM// Whether to reload consuming segment on scheme update. Will change default behavior to true when this feature is stabilizedI assume this has happened already
II
06/23/2021, 8:29 PMQianbo Wang
06/23/2021, 8:51 PMgroup by
statement? thanks in advance.Juraj Pohanka
06/24/2021, 2:37 PMJai Patel
06/24/2021, 5:03 PMNeil Teng
06/24/2021, 6:51 PM