apache-pinot #troubleshooting

Hello, does Pinot support some auto-scaling of some sort to deal with increasingly larger and heavier workloads? I have a single real-time table consuming events from kafka (this is a wide table but not many fields, currently there are 39, mostly strings, one of which has a max length of 2147483647 (INTEGER.MAX_VALUE) since it holds a json blob). My pinot cluster is deployed in Kubernetes (hosted in azure) 2 pinot server instances with 5GB heap + 3GB for direct memory, 100GB persistance volume (segment deepstorage is configured) with a k8s memory limit of 10G. 1 controller instance with 1GB heap for JVM, k8s memory limit 2G. 1 broker instance with 4GB heap, k8s memory limit 5G. My servers are crashing with segment faults & OOM, as follows: Server 1:

Copy code

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f4b79052422, pid=8, tid=0x00007f4ae8739700
#
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jbyte_disjoint_arraycopy
#
# Core dump written. Default location: /opt/pinot/core or core.8
#
[thread 139959708407552 also had an error]
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid8.log
#
# If you would like to submit a bug report, please visit:
#   <http://bugreport.java.com/bugreport/crash.jsp>
#
Aborted (core dumped)

Server 2:

Copy code

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)"
2021/06/08 16:35:05.338 ERROR [LLRealtimeSegmentDataManager_HitExecutionView_3mo__1__3__20210608T1552Z] [HitExecutionView_3mo__1__3__20210608T1552Z] Could not build segment
java.lang.IllegalArgumentException: Self-suppression not permitted
	at java.lang.Throwable.addSuppressed(Throwable.java:1072) ~[?:1.8.0_292]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:132) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:783) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:717) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:628) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.lang.OutOfMemoryError: Java heap space
AsyncLogger error handling event seq=1, value='null': java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "HitExecutionView_3mo__3__3__20210608T1552Z" java.lang.OutOfMemoryError: Java heap space
2021/06/08 16:35:05.395 ERROR [LLRealtimeSegmentDataManager_HitExecutionView_3mo__7__3__20210608T1553Z] [HitExecutionView_3mo__7__3__20210608T1553Z] Could not build segment
java.lang.IllegalArgumentException: Self-suppression not permitted
	at java.lang.Throwable.addSuppressed(Throwable.java:1072) ~[?:1.8.0_292]
	at org.apache.pinot.segment.local.segment.index.converter.SegmentV1V2ToV3FormatConverter.copyIndexData(SegmentV1V2ToV3FormatConverter.java:160) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.segment.local.segment.index.converter.SegmentV1V2ToV3FormatConverter.convert(SegmentV1V2ToV3FormatConverter.java:86) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.convertFormatIfNecessary(SegmentIndexCreationDriverImpl.java:370) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:303) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:256) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:783) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:717) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:628) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.lang.OutOfMemoryError: Java heap space

Sadim Nadeem

06/09/2021, 7:39 AM

is there any way to aggreagete time series data on basis of tumbling time window while ingesting in pinot itself say on basis of time and update the records automatically without any need of tumbling window aggregation in my stream processing job(ex-samza job/spark streaming) .. means can inot keep updating the same countPerMin column automatically whenever new feeds for same time window comes .. or they have to be handled in samza jobs/spark streaming only .cc: @Mohamed Hussain

Map

06/09/2021, 5:04 PM

Hi Folks, it seems that I cannot get

docker build

to work although it compiles on my local computer. For example, if I do

./docker-build.sh pinot

under

docker/images/pinot/docker-build.sh

, it fails when compiling

pinot-controller

, and the error is

Copy code

#12 241.2 [INFO] --- frontend-maven-plugin:1.1:npm (npm run-script build) @ pinot-controller ---
#12 241.2 [INFO] Running 'npm run-script build' in /opt/pinot-build/pinot-controller/src/main/resources
#12 241.4 [INFO] 
#12 241.4 [INFO] > pinot-controller-ui@1.0.0 build /opt/pinot-build/pinot-controller/src/main/resources
#12 241.4 [INFO] > webpack --mode production
#12 241.4 [INFO] 
#12 257.6 [ERROR] Killed
#12 257.6 [ERROR] npm ERR! code ELIFECYCLE
#12 257.6 [ERROR] npm ERR! errno 137
#12 257.6 [ERROR] npm ERR! pinot-controller-ui@1.0.0 build: `webpack --mode production`
#12 257.6 [ERROR] npm ERR! Exit status 137
#12 257.6 [ERROR] npm ERR! 
#12 257.6 [ERROR] npm ERR! Failed at the pinot-controller-ui@1.0.0 build script.
#12 257.6 [ERROR] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
#12 257.6 [ERROR] 
#12 257.6 [ERROR] npm ERR! A complete log of this run can be found in:
#12 257.6 [ERROR] npm ERR!     /root/.npm/_logs/2021-06-09T16_55_54_192Z-debug.log
...
#12 264.0 [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.1:npm (npm run-script build) on project pinot-controller: Failed to run task: 'npm run-script build' failed. (error code 137) -> [Help 1]
#12 264.0 [ERROR] 
#12 264.0 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
#12 264.0 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
#12 264.0 [ERROR] 
#12 264.0 [ERROR] For more information about the errors and possible solutions, please read the following articles:
#12 264.0 [ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException>
#12 264.0 [ERROR] 
#12 264.0 [ERROR] After correcting the problems, you can resume the build with the command
#12 264.0 [ERROR]   mvn <args> -rf :pinot-controller

Am I missing something obvious? Not sure if someone else can reproduce it.

Phúc Huỳnh

06/10/2021, 10:57 AM

I got some problem with query result. I run the same query statement

Copy code

select count(*) from RuleLogsUAT

But got timeout & difference result per request. Here is the request log

Copy code

requestId=43,table=RuleLogsUAT,timeMs=10001,docs=210279/210377,entries=0/0,segments(queried/processed/matched/consuming/unavailable):21/18/18/2/0,consumingFreshnessTimeMs=1623319344461,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-server-1_R=1,-1,0,0;pinot-server-2_R=2,6,250,0;pinot-server-0_R=2,6,250,0;pinot-server-0_O=1,7,230,0;pinot-server-1_O=2,11,230,0,query=select count(*) from RuleLogsUAT,offlineThreadCpuTimeNs=1548682,realtimeThreadCpuTimeNs=1908620
requestId=44,table=RuleLogsUAT,timeMs=10002,docs=44807/44811,entries=0/0,segments(queried/processed/matched/consuming/unavailable):19/17/17/3/0,consumingFreshnessTimeMs=1623319158004,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-server-1_R=1,10,250,0;pinot-server-2_R=1,7,250,0;pinot-server-0_R=1,8,250,0;pinot-server-2_O=1,-1,0,0;pinot-server-1_O=1,11,230,0,query=select count(*) from RuleLogsUAT,offlineThreadCpuTimeNs=1211610,realtimeThreadCpuTimeNs=1412400
requestId=45,table=RuleLogsUAT,timeMs=10002,docs=73633/73731,entries=0/0,segments(queried/processed/matched/consuming/unavailable):23/20/20/3/0,consumingFreshnessTimeMs=1623319158007,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-server-1_R=0,5,250,0;pinot-server-2_R=0,8,250,0;pinot-server-0_R=0,8,250,0;pinot-server-0_O=0,7,230,0;pinot-server-1_O=1,-1,0,0,query=select count(*) from RuleLogsUAT,offlineThreadCpuTimeNs=655059,realtimeThreadCpuTimeNs=1141471
requestId=46,table=RuleLogsUAT,timeMs=10002,docs=44807/44811,entries=0/0,segments(queried/processed/matched/consuming/unavailable):19/17/17/3/0,consumingFreshnessTimeMs=1623319158004,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-server-1_R=1,5,250,0;pinot-server-2_R=1,7,250,0;pinot-server-0_R=1,6,250,0;pinot-server-2_O=1,-1,0,0;pinot-server-1_O=1,7,230,0,query=select count(*) from RuleLogsUAT,offlineThreadCpuTimeNs=417430,realtimeThreadCpuTimeNs=1821230

Shailesh Jha

06/10/2021, 12:21 PM

Hi @Mayank @Neha Pawar @Daniel Lavoie We have segments backed-up in GCS. How can we reuse that segment files?? How backup and restore works?? Do we have any documentation for this? CC: @Sadim Nadeem @Mohamed Sultan

Ravi R

06/10/2021, 12:32 PM

Hello, Setting up pinot on a docker-compose container. Everthings runs as expected but when try to create a stream table by using the below commands getting Connection refused error.

Copy code

bin/pinot-admin.sh AddTable \
    -schemaFile examples/stream/airlineStats/airlineStats_schema.json \
    -tableConfigFile examples/stream/airlineStats/airlineStats_realtime_table_config.json \
    -exec

Error:

Copy code

Got Exception to upload Pinot Schema: airlineStats
shaded.org.apache.http.conn.HttpHostConnectException: Connect to 172.26.0.5:9000 [/172.26.0.5] failed: Connection refused (Connection refused)
	at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:383) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.common.utils.FileUploadDownloadClient.addSchema(FileUploadDownloadClient.java:448) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.AddTableCommand.uploadSchema(AddTableCommand.java:135) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.AddTableCommand.execute(AddTableCommand.java:170) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at shaded.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	... 15 more
Exception caught: 
shaded.org.apache.http.conn.HttpHostConnectException: Connect to 172.26.0.5:9000 [/172.26.0.5] failed: Connection refused (Connection refused)
	at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:383) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.common.utils.FileUploadDownloadClient.addSchema(FileUploadDownloadClient.java:448) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.AddTableCommand.uploadSchema(AddTableCommand.java:135) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.AddTableCommand.execute(AddTableCommand.java:170) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at shaded.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	... 15 more

Copy code

DOCKER_HOST=unix:///var/run/docker.sock docker ps

Copy code

CONTAINER ID   IMAGE                             COMMAND                  CREATED       STATUS       PORTS                                                                          NAMES
545bf572fd0d   apachepinot/pinot:release-0.7.1   "./bin/pinot-admin.s…"   2 hours ago   Up 2 hours   8096-8097/tcp, 8099/tcp, 9000/tcp, 0.0.0.0:8098->8098/tcp, :::8098->8098/tcp   composetest_pinot-server_1
4033365f967e   apachepinot/pinot:release-0.7.1   "./bin/pinot-admin.s…"   2 hours ago   Up 2 hours   8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp             composetest_pinot-broker_1
1712fba17460   apachepinot/pinot:release-0.7.1   "./bin/pinot-admin.s…"   2 hours ago   Up 2 hours   8096-8099/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                       composetest_pinot-controller_1
aeac35e847b6   confluentinc/cp-kafka:5.3.0       "/etc/confluent/dock…"   2 days ago    Up 2 hours   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                      composetest_kafka_1
050bf6511ab9   zookeeper:latest                  "/docker-entrypoint.…"   2 days ago    Up 2 hours   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp        composetest_zookeeper_1

Can anyone kindly suggest how to proceed from here ?

Aaron Wishnick

06/10/2021, 4:35 PM

I'm seeing that pinot-admin.sh will error out but not return a nonzero return code, e.g. when data ingestion job specs are malformed

Jonathan Meyer

06/10/2021, 5:33 PM

Hello 🙂 Are subqueries supported in Pinot ? Such as 1.

SELECT * FROM table WHERE x in (SELECT ...)

and 2.

SELECT * FROM (SELECT ...)

(i.e. working with the output of a subquery)

luanmorenomaciel

06/10/2021, 7:14 PM

hi fellows, i'm trying to deploy the latest version of pinot on my k8s but getting this funky issue, is there someone to shed some lights?

Copy code

Starting a Pinot [SERVICE_MANAGER] at 0.08s since launch
Registering service status handler
Skip Starting Pinot Service Manager admin application
Started Pinot [SERVICE_MANAGER] instance [ServiceManager_pinot-broker-0.pinot-broker-headless.datastore.svc.cluster.local_-1] at 0.084s since launch
Starting a Pinot [BROKER] at 0.087s since launch
Trying to start Pinot Broker...
Failed to initialize Pinot Broker Starter
java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_292]
	at java.lang.Integer.parseInt(Integer.java:592) ~[?:1.8.0_292]
	at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_292]
	at org.apache.pinot.core.util.ListenerConfigUtil.buildBrokerConfigs(ListenerConfigUtil.java:107) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.broker.broker.helix.HelixBrokerStarter.<init>(HelixBrokerStarter.java:131) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:133) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:259) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:285) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:56) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:259) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
Failed to start a Pinot [BROKER] at 0.114 since launch

Calvin Karundu

06/11/2021, 9:52 AM

Hey guys, what’s the best way - if any - to deal with JSON queries where 1) Some data points might be null 2) Not all data points have the same schema … here’s some sample data:

Copy code

Pinot schema column is called "properties" with a type of string

# First Record
{
    timestamp: '...',
    browser: '...'
}

# Second Record
null

# Third Record
{
    balance: '...'
}

This data is provided by the end user and can be whatever they’d like (this is by design) … so there’s really no “schema” for the data in the JSON column.

Jonathan Meyer

06/13/2021, 10:59 AM

Hello Is there a way to use lookups (dimTable) inside a WHERE clause ? Something like

SELECT SUM(value) FROM table WHERE user IN LOOKUP('group', 'user', 'groupId', '<groupId>')

? (which isn't valid) Basically my goal is to fetch a list of 'users' in the dimTable using a 'group' identifier and filtering on those (in the main table)

luanmorenomaciel

06/14/2021, 5:59 PM

hi experts, i've a running realtime table that gets data from kafka running, today when I checked for new incoming data, i've got this error, any ideas what that could be?

Copy code

21/06/11 21:08:15.665 ERROR [LLRealtimeSegmentDataManager_realtime_enriched_music_data_users__7__0__20210611T2008Z] [realtime_enriched_music_data_users__7__0__20210611T2008Z] Could not build segment
java.lang.IllegalArgumentException: Invalid format: "2021-06-10 10:42:25" is too short
	at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.writeMetadata(SegmentColumnarIndexCreator.java:552) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:512) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:284) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:257) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:794) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:728) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:634) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2021/06/11 21:08:15.665 ERROR [LLRealtimeSegmentDataManager_realtime_enriched_music_data_users__4__0__20210611T2008Z] [realtime_enriched_music_data_users__4__0__20210611T2008Z] Could not build segment
java.lang.IllegalArgumentException: Invalid format: "2021-06-10 10:42:25" is too short
	at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.writeMetadata(SegmentColumnarIndexCreator.java:552) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:512) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:284) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]

Elon

06/14/2021, 10:52 PM

Hi, we just experienced an issue where a server restarted, and when downloading a segment from gcs it threw an

java.io.IOException: Input is not in the .gz format

- we saw that the segment was just being written to gcs. Once I deleted the segment on the server and restarted it downloaded it without any issues. Has anyone ever experienced that before? I can create a github issue w some ideas for fixes...

khush

06/15/2021, 6:23 AM

Hi, I have followed this documentation to query pinot using Java client: https://docs.pinot.apache.org/users/clients/java I have scheduled the query to run after every 24 hours. The first run goes fine. However the subsequent iteration fails with the below exception. 2021-06-15 052309.880 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) 2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session 2021-06-15 052309.911 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052310.279 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) Any help here?

Azri Jamil

06/15/2021, 2:39 PM

Hi expert, I just add a geospatial index on existing realtime table, the new data ingested generate the index correctly.. but how do I apply it for previous segment as well?

Surendra

06/15/2021, 5:38 PM

Hi, We have below query working in 0.6.0 and failing 0.7.1 with exception

2021/06/10 16:53:02.084 WARN [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-100] Caught exception while updating Column names in Query 13040:  Illegal AstNode type for TransformExpressionTree: org.apache.pinot.pql.parsers.pql2.ast.StarExpressionAstNode

and if we update

else null

in case clause to

else 'null'

it's working in .7.1 , any clues ?

Copy code

select emp_id ,
id ,
max(timestampMs) as last_updated,
count(DISTINCT  case when emp_type  = 'admin' then user_count else null end) as admin_counts
from emp_events
where org_id = 987
AND timestampMs >= 1620892156939
AND timestampMs < 1621630122162
group by emp_id, id
limit 10

Jonathan Meyer

06/15/2021, 6:45 PM

Hello ^^ Not really an issue, just checking if this is "normal behavior"

select DISTINCT(kpi) from kpis

takes ~6ms (with 100M docs, &

numDocsScanned: 100000

) - this query returns 45 strings only But doing

select DISTINCT(kpi) from kpis ORDER BY kpi

takes >300ms (50 times slower) - It scans every documents (

numDocsScanned: 101250000

) I guess the

ORDER BY

breaks some optimizations down But from the outside it seems like pretty surprising behavior (sorting 45 strings "should not take this long" is what I mean) Anyway, not here to complain, just wanted to point it out in case it would be considered as something worth investigating

Jai Patel

06/15/2021, 8:41 PM

I’m trying to setup a dateTimeFieldSpec with a dateTrunc and I’m getting an error:

Copy code

{
      "name": "created_at_1_week_seconds",
      "dataType": "LONG",
      "defaultNullValue": 0,
      "transformFunction": "dateTrunc('week', created_at, 'MILLISECONDS')",
      "format": "1:SECONDS:EPOCH",
      "granularity": "1:SECONDS"
    },

Error:

Copy code

Exception in getting arguments for transform function 'dateTrunc('week', created_at, MILLISECONDS)' for column 'created_at_1_week_seconds'"
}

This works:

Copy code

{
      "name": "created_at_1_day_seconds",
      "dataType": "LONG",
      "defaultNullValue": 0,
      "transformFunction": "toEpochSecondsRounded(fromEpochDaysBucket(toEpochDaysBucket(created_at, 1), 1), 1)",
      "format": "1:SECONDS:EPOCH",
      "granularity": "1:SECONDS"
    },

Something about the dateTrunc function doesn’t seem to be compatible. Looking through the docs and the code it looks like dateTrunc may not be listed as a valid transform function in all use cases. The main issue is I want a toEpochDaysBucket(7) that lines up with the calendar week (Sunday or Monday) instead of the epoch week (Thursday). Any ideas?

Surendra

06/15/2021, 11:47 PM

What are the different options available to delete tables/segments ? Using a API it's taking lot of time as we have > 20k segments.

Eugene Ramirez

06/16/2021, 3:53 AM

Hi, I am evaluating Pinot for possible production use in my company. I am encountering problem on

back up/restore

feature. I appreciate if anyone can help. Here is my setup. Kubernetes: EKS 1.20.4 Pinot version: 0.7.1 So I enable S3 as deep storage based from this link. Then ingested Parquet data from S3 based on this instruction. Data loaded fine and I can query the expected data from Pinot. Next I simulated replacing the cluster, by uninstalling all pods and its related volumes(therefore losing all state) but kept the segment files in s3 segment location(therefore backup is intact in deep store). Next I reinstalled cluster, and reconfigured the tables. I was expecting that the servers would automatically fetch the segments from deep store as mentioned in previous post, but it does not seem to be happening. Am i missing a step? Thanks in advance. https://apache-pinot.slack.com/archives/C011C9JHN7R/p1623336369017900?thread_ts=1623327667.015000&cid=C011C9JHN7R

jainendra tarun

06/16/2021, 4:59 PM

Hello everyone, I am running Presto to query Pinot and the presto-pinot connector throws an exception when there is no data returned by Pinot. Example query is :

Copy code

select * from pinot.default.mytable where where datekey='2021-04-19 00:00:00' limit 10

Though this query returns empty result in Pinot as expected. The exception in presto is :

Copy code

java.lang.IllegalStateException: Expected at least one row to be present
	at com.google.common.base.Preconditions.checkState(Preconditions.java:507)
	at com.facebook.presto.pinot.PinotBrokerPageSourceSql.populateFromQueryResults(PinotBrokerPageSourceSql.java:118)
	at com.facebook.presto.pinot.PinotBrokerPageSourceBase.lambda$issueQueryAndPopulate$0(PinotBrokerPageSourceBase.java:327)
	at com.facebook.presto.pinot.PinotUtils.doWithRetries(PinotUtils.java:39)
	at com.facebook.presto.pinot.PinotBrokerPageSourceBase.issueQueryAndPopulate(PinotBrokerPageSourceBase.java:312)
	at com.facebook.presto.pinot.PinotBrokerPageSourceBase.getNextPage(PinotBrokerPageSourceBase.java:222)
	at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:252)
	at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
	at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
	at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
	at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
	at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
	at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
	at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599)
	at com.facebook.presto.$gen.Presto_0_256_SNAPSHOT_5059796____20210616_162510_1.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Is this a bug or am I missing some configuration ?

06/16/2021, 5:35 PM

@ Hi Team, I am ingesting realtime data from Kafka and updating realtime data in superset dashboard. In superset dashboard I have one slice where I am displaying events of last 5 min based on my timestamp columns. So for testing purpose I have pushed one event in Kafka which was already available ( duplicate) as soon as pushed the data in Kafka it's showing on pinot within milisec.but same it's not reflecting at dashboard side as last 5 mins count. So my question is. Will it take sometime to reflect at dashboard side ?or duplicates records will not show as last 5 mins. Count at Dashboard. @Mayank

Mateus Oliveira

06/16/2021, 7:53 PM

Hello team, need helo with something, I'm trying to load some data from S3 bucket into Pinot but is give me this error

Copy code

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme s3, classname org.apache.pinot.plugin.filesystem.S3PinotFS
Creating an executor service with 1 threads(Job parallelism: 0, available cores: 1.)
Listed 8 files from URI: <s3://landing/bank/>, is recursive: true
Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:144) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:113) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:132) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
Caused by: java.lang.IllegalArgumentException
	at sun.nio.fs.UnixFileSystem.getPathMatcher(UnixFileSystem.java:288) ~[?:1.8.0_292]
	at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:175) ~[pinot-batch-ingestion-standalone-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:142) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
	... 4 more

this is my job

Copy code

executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '<s3://landing/bank/>'
includeFileNamePattern: '*.json'
outputDirURI: '<s3://pinot/>'
overwriteOutput: true
pinotFSSpecs:
    - scheme: s3
      className: org.apache.pinot.plugin.filesystem.S3PinotFS
      configs:
        region: 'us-east-1'
        endpoint: '<http://10.0.220.205:9000>'
        accessKey: 'access'
        secretKey: 'key'
recordReaderSpec:
    dataFormat: 'json'
    className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader'
tableSpec:
    tableName: 'bank'
pinotClusterSpecs:
    - controllerURI: '<http://localhost:9000>'

Jackie

06/17/2021, 1:46 AM

It doesn’t support connection pool, but Pinot supports jdbc connector. @Xiang Fu can you share more info about the jdbc connector?

Daniel Lavoie

06/17/2021, 2:03 AM

Pinot is a rest API that can support high QPS, what benefits are you looking into a connection pool? Connection pooling would likely be a threading configuration from your java service http client.

Laxman Ch

06/18/2021, 2:50 PM

Hi All, can someone please point me to some detailed documentation on metric aggregation in Pinot. Documentation I found on this is very limited. I’m looking for following information. • Does REALTIME tables support aggregation/rollup during ingestion? • What are the different types of aggregation types supported (max, min, sum and anymore?)? • Any known limitations in using aggregations in REALTIME & OFFLINE tables? • Any general best practices and gotchas with aggregations/rollups?

Jai Patel

06/18/2021, 4:51 PM

Good morning (in Seattle) folks. i wanted some help troubleshooting a Pinot (0.6.0) upsert table. For context: 1. This table was deployed to our staging environment and production environment. Exact same schema and tablespec. Works fine in staging streaming junk data. Not so much in production on real data. 2. Retention time is 10 days. 3. After periods of idleness, we are seeing cases where the production instance returning no data. Try again 10 minutes later and everything is fine. 4. Querying for age of the newest record, it’s about 2 minutes old in production. Which seems right. 5. Some observations I noticed: a. Our time column (processed_at) is not the same as our sorted column index (created_at_seconds) b. We are on Pinot 0.6.0 (old bug?) c. We have only two upsert tables like this providing different views of the data on the cluster. d. The cluster is resourced for “testing.” Does Pinot evict idle tables out of memory? Could it be slow to reload it because of the index? Is it the resources? Is there a known bug I’m htiting? cc: @Elon @Xiang Fu @Neha Pawar

Aaron Wishnick

06/18/2021, 8:58 PM

Any suggestion for speeding up a query that uses REGEX_LIKE to filter on a dimension? I see string operations being super slow. Even if I rewrite my regex as

SUBSTR(foo, ..., ...) = bar

I still see the query taking more than 10 seconds

Zsolt Takacs

06/21/2021, 7:08 AM

We are having trouble with RealtimeToOfflineSegmentsTask, doing rollups fills up the heap and causes bad GC stalls disconnecting ZK and rendering the minion useless until a restart. Since the task is time based it's not possible to size the minion heap to account for traffic spikes without overshooting towards too small segments When is the off-heap rollup support planned to be implemented? (mentioned in https://github.com/apache/incubator-pinot/pull/6847)

Jonathan Meyer

06/21/2021, 7:25 AM

Hello 🙂 Can someone remind me the steps to handle

"message": "MergeResponseError: Data schema mismatch between merged block:

after a REALTIME table backward-compatible schema change ? (new column) I've tried

Reload All Segments

Is that because I haven't set pinot.server.instance.reload.consumingSegment
?