HongChe Lin
04/20/2023, 9:22 AMSleeping 1 second waiting for all segments loaded for table: ...
I also found lots of logs in controller said Event b45f4423_DEFAULT : Message already exists for {server} to transit {table}.{segment} from OFFLINE to ONLINE, isRelay: false
The cpu and memory resource did not exhaust by controller and zookeeper. How can I speed up segment reload?Shreeram Goyal
04/21/2023, 8:22 AMShubham Kumar
04/21/2023, 8:40 AMjava.sql.SQLException: Cannot create PoolableConnectionFactory (null)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:653) ~[commons-dbcp2-2.9.0.jar:2.9.0]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:531) ~[commons-dbcp2-2.9.0.jar:2.9.0]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731) ~[commons-dbcp2-2.9.0.jar:2.9.0]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.initPool(DataSourceElement.java:308) ~[ApacheJMeter_jdbc.jar:5.5]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.testStarted(DataSourceElement.java:128) ~[ApacheJMeter_jdbc.jar:5.5]
at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:218) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:396) ~[ApacheJMeter_core.jar:5.5]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: java.sql.SQLFeatureNotSupportedException
at org.apache.pinot.client.base.AbstractBaseConnection.getAutoCommit(AbstractBaseConnection.java:114) ~[pinot-jdbc-client-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-0797e05da61c37629da3e99bdeb4d96dc60d7911]
at org.apache.commons.dbcp2.DelegatingConnection.getAutoCommit(DelegatingConnection.java:308) ~[commons-dbcp2-2.9.0.jar:2.9.0]
at org.apache.commons.dbcp2.PoolableConnectionFactory.activateObject(PoolableConnectionFactory.java:119) ~[commons-dbcp2-2.9.0.jar:2.9.0]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:108) ~[commons-dbcp2-2.9.0.jar:2.9.0]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649) ~[commons-dbcp2-2.9.0.jar:2.9.0]
... 7 more
can someone please help on how to establish connection properly in jmeterPratik Bhadane
04/21/2023, 10:35 AMChris Han
04/21/2023, 12:28 PMhelm install --set config.key=value
parth
04/21/2023, 2:30 PM...IS NOT NULL
condition, the query fails (trino: internal error) , but when the query uses left / right join with the same conditions, it executes albeit with incorrect results. Any ideas on why this may be happening and how we should go about debugging it?
Thanks !!!Hassan Ait Brik
04/21/2023, 2:52 PMsuraj sheshadri
04/22/2023, 5:41 AMTanmay Varun
04/22/2023, 4:31 PMpramod shenoy
04/24/2023, 4:44 AMChris Han
04/24/2023, 9:23 AMChris Han
04/24/2023, 11:53 AMCaused by: java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized
exactly mean? Log is from the controller. Assuming I passed in in the correct configs to helm values, do I need to do a custom build of Pinot?Raveendra Yerraguntla
04/24/2023, 1:10 PMUtkarsh
04/24/2023, 6:16 PMPROPERTYSTORE/SEGMENTS/<table_name>
path of a cluster's zookeeper? ( I am looking for to use swagger for this ) .Malte Granderath
04/25/2023, 11:40 AMChannelHandlerContext
inside the controller and broker access control implementations? Looking at the code probably not but just wanted to make sure. Ideally we would want to use mTLS in combination with the information stored in the cert to make decisions whether a user has access to an operationsuraj sheshadri
04/25/2023, 11:05 PMZhengfei
04/26/2023, 7:32 AMSparkSegmentTarPushJobRunner
to create and push segment files to pinot
[HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread_22] Exception while executing a state transition task dimension_xxx
java.lang.reflect.InvocationTargetException: null
at jdk.internal.reflect.GeneratedMethodAccessor356.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.RuntimeException: Caught exception while loading lookup table: xxxDim_OFFLINE after adding segment: dimension_xxx
at org.apache.pinot.core.data.manager.offline.DimensionTableDataManager.addSegment(DimensionTableDataManager.java:101) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.BaseTableDataManager.addSegment(BaseTableDataManager.java:216) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:409) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:385) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:163) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
... 11 more
Caused by: java.lang.RuntimeException: Failed to use PinotSegmentRecordReader to read immutable segment
at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.getRecord(ImmutableSegmentImpl.java:232) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.offline.DimensionTableDataManager.createDimensionTable(DimensionTableDataManager.java:146) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.offline.DimensionTableDataManager.loadLookupTable(DimensionTableDataManager.java:127) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.offline.DimensionTableDataManager.addSegment(DimensionTableDataManager.java:97) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.BaseTableDataManager.addSegment(BaseTableDataManager.java:216) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:409) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:385) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:163) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
Could you please help take a look?Gunvant Patil
04/26/2023, 7:53 AMAlexander Vivas
04/26/2023, 10:25 AMparth
04/26/2023, 3:26 PM"realtime.segment.flush.threshold.rows":"10000",
"realtime.segment.flush.threshold.time":"2m"
and im pushing messages to kafka. i want the consuming segment to commit and ive tried pushing data with timestamps 5 minutes apart. even pushed messages with timestamp 5 minutes greater than the "segment.creation.time". But the consuming segment still stays open.
Please help me understand this behaviour.Phil Sheets
04/27/2023, 2:49 PMNo enum constant java.util.concurrent.TimeUnit.milliseconds
For this transformation:
{
"columnName": "device_timestamp_hour_bucket",
"transformFunction": "dateTrunc('hour', fromDateTime(deviceTimestamp, 'yyyy-MM-dd''T''HH:mm:ss.SSS''Z'), 'milliseconds')"
},
It works fine when I run it in the query explorerNeeraja Sridharan
04/27/2023, 4:49 PMRaveendra Yerraguntla
04/27/2023, 9:42 PMChris Han
04/28/2023, 11:03 AMhimanshu yadav
04/28/2023, 1:09 PMselect * from test_table;
Query 20230428_130204_00001_43kjv failed: java.net.UnknownHostException: pinot-broker-1.pinot-broker-headless.default-namespace.svc.cluster.local: Name or service not known
how can pass the broker service endpoint to presto pinot connector config.
current config of pinot
connector.name=pinot
pinot.controller-rest-service=<pinot-controller-url>
pinot.controller-urls=<pinot-controller-url>
pinot.secure-connection=true
pinot.rest-proxy-service-for-query=true
pinot.allow-multiple-aggregations=true
pinot.use-date-trunc=true
pinot.infer-date-type-in-schema=true
pinot.infer-timestamp-type-in-schema=true
pinot.use-streaming-for-segment-queries=false
pinot.forbid-broker-queries=true
Raveendra Yerraguntla
04/29/2023, 10:35 AMadheip singh
04/30/2023, 8:00 PMChris Han
05/01/2023, 2:27 PMTiger Zhao
05/01/2023, 6:09 PMDeleted_Segments/
folder for the controller, and it appears that the data in this folder is not actually being deleted and is building up over time. Just wanted to check if there are any known bugs around this? And if it is safe to manually the delete the data in the folder? (note that this is for the controller deepstore using S3). I see a large number of left over segments that were supposed to be deleted due to retention set on the tables.Idan Freundlich
05/02/2023, 8:30 AMProcessingException(errorCode:150, message:PQLParsingError:
java.lang.ClassCastException: class org.apache.calcite.sql.SqlInsert cannot be cast to class org.apache.calcite.sql.SqlSelect (org.apache.calcite.sql.SqlInsert and org.apache.calcite.sql.SqlSelect are in unnamed module of loader 'app')
at org.apache.pinot.sql.parsers.CalciteSqlParser.compileSqlNodeToPinotQuery(CalciteSqlParser.java:357)
at org.apache.pinot.sql.parsers.CalciteSqlParser.compileToPinotQuery(CalciteSqlParser.java:143)
at org.apache.pinot.sql.parsers.CalciteSqlCompiler.compileToBrokerRequest(CalciteSqlCompiler.java:35)
at org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java:166))
Is there other way that I can insert data not via kafka or s3?