https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
l

Luis Fernandez

04/07/2022, 8:25 PM
in the pinot docs we have this about start pinot via IDE but how is this starting the pinot infra via the IDE? I guess that ultimately my question is how can I attach remote debugger to my local pinot processes
m

Mayank

04/07/2022, 8:40 PM
It is suggesting to start the
quickStart
program, which internally starts all pinot components within the same jvm. You can run Pinot and debug it in IDE as you do any application.
l

Luis Fernandez

04/07/2022, 8:48 PM
i got this exception do yo uknow why it may be?
Copy code
Instance 0.0.26.108_9000 is not leader of cluster QuickStartCluster due to exception happen when session check
org.I0Itec.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException
i was trying to run the empty quick start model https://github.com/apache/pinot/pull/8024
m

Mayank

04/07/2022, 9:37 PM
This is a newer feature. @Kenny Bastani any idea on what might be going on?
k

Kenny Bastani

04/07/2022, 9:38 PM
@Luis Fernandez What command are you using to start Pinot?
l

Luis Fernandez

04/07/2022, 9:40 PM
like this sh pinot-admin.sh QuickStart -type EMPTY -dataDir “usr/local/var/lib/pinot/data”
k

Kenny Bastani

04/07/2022, 9:41 PM
Do you have ZK running externally?
l

Luis Fernandez

04/07/2022, 9:42 PM
no
but i also don’t have zookeeper running locally do i have to run zk manually first i thought this would start zk for me
k

Kenny Bastani

04/07/2022, 9:43 PM
Yes, it will
One sec
Please run this command
netstat -vanp tcp | grep '*.2123\|9000\|8000\|7000'
And let me know what the output is
Also,
ls /usr/local/var/lib/pinot/data/rawdata
@Luis Fernandez Let me know if you got it solved. Happy to jump on a call if you need help with anything.
l

Luis Fernandez

04/08/2022, 1:49 PM
hey Kenny sorry, was out yesterday already, that netstat command is while i’m trying to run the command yes?
also i’m running this from the source code itself, so i’m running from
pinot/build
don’t know if that has anything to do
this is the output of the first command you put
Copy code
tcp6       0      0  *.7000                 *.*                    LISTEN      131072 131072   1905      0 0x0100 0x00000006
tcp4       0      0  *.7000                 *.*                    LISTEN      131072 131072   1905      0 0x0100 0x00000006
oh i’m dumb hehe it was airplay receiver it’s on the pinot docs thank you both for your help i think we gucci
👍 1
@Mayank when i saw the title running from IDE I was thinking that it would literally start off from the IDE like run everything from intellij for example, do you know who may know how to setup debugging for this? I have it running but i’m unsure as to how to debug
m

Mayank

04/08/2022, 6:21 PM
Yes, you can literally start off from IDE. You can pick one of the quick start classes and hit the green run button in the main function from within IDE.
l

Luis Fernandez

04/08/2022, 7:54 PM
anyone familiar with this error
Copy code
2022/04/08 15:53:37.705 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1
java.net.NoRouteToHostException: No route to host (Host unreachable)
anyone familiar with this error while executing the QuickStart from IDE
Copy code
2022/04/08 15:53:37.705 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1
java.net.NoRouteToHostException: No route to host (Host unreachable)
m

Mayank

04/08/2022, 10:27 PM
Is broker up and running at port 8000 of 0.0.26.108?
l

Luis Fernandez

04/09/2022, 12:00 AM
when i execute queries i get that error
Copy code
2022/04/08 20:00:59.462 INFO [StartBrokerCommand] [main] Executing command: StartBroker -brokerHost null -brokerPort 8000 -zkAddress localhost:2123
Copy code
2022/04/08 20:01:00.457 INFO [StartServiceManagerCommand] [Start a Pinot [BROKER]] Started Pinot [BROKER] instance [Broker_0.0.26.108_8000] at 9.464s since launch
k

Kenny Bastani

04/09/2022, 4:45 PM
Why is your IP not localhost?
l

Luis Fernandez

04/09/2022, 4:53 PM
That's an amazing question 😂
m

Mayank

04/09/2022, 11:46 PM
Is it not the local host’s IP?
l

Luis Fernandez

04/11/2022, 2:12 PM
0.0.26 i guess it’s good
but i don’t understand what that error is about
is there anything in particular i have to do if i’m running in mac?
m

Mayank

04/11/2022, 2:26 PM
Is it M1 or are you on vpn?
l

Luis Fernandez

04/11/2022, 2:26 PM
that’s the processor yes?
I’m on vpn yes
i’m not on m1 but def have vpn on at the moment
m

Mayank

04/11/2022, 2:28 PM
Try with vpn disconnected
My guess is that it is messing up the connectivity
l

Luis Fernandez

04/11/2022, 2:29 PM
aw i still see the error
Copy code
java.net.NoRouteToHostException: No route to host (Host unreachable)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/java.net.Socket.connect(Socket.java:558)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1248)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1337)
	at org.apache.pinot.tools.admin.command.AbstractBaseAdminCommand.sendRequest(AbstractBaseAdminCommand.java:91)
	at org.apache.pinot.tools.admin.command.PostQueryCommand.run(PostQueryCommand.java:145)
	at org.apache.pinot.tools.admin.command.QuickstartRunner.runQuery(QuickstartRunner.java:280)
	at org.apache.pinot.tools.Quickstart.runSampleQueries(Quickstart.java:192)
	at org.apache.pinot.tools.Quickstart.execute(Quickstart.java:130)
	at org.apache.pinot.tools.admin.command.QuickStartCommand.execute(QuickStartCommand.java:157)
	at org.apache.pinot.tools.Command.call(Command.java:33)
	at org.apache.pinot.tools.Command.call(Command.java:29)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:161)
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192)
	at org.apache.pinot.tools.Quickstart.main(Quickstart.java:229)
there’s also this
Copy code
2022/04/11 10:29:11.620 ERROR [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread] Attempts exceeded when downloading segment: baseballStats_OFFLINE_0 for table: baseballStats_OFFLINE from: <http://0.0.26.108:9000/segments/baseballStats/baseballStats_OFFLINE_0> to: /Users/luisfernandez/development/pinot/usr/local/var/lib/pinot/data/1649687329197/baseballStats/rawdata/PinotServerDataDir0/baseballStats_OFFLINE/tmp/tmp-baseballStats_OFFLINE_0-edf388f0-9451-4948-8730-a6cf9987103e/baseballStats_OFFLINE_0.tar.gz
2022/04/11 10:29:11.624 ERROR [SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread] Caught exception in state transition from OFFLINE -> ONLINE for resource: baseballStats_OFFLINE, partition: baseballStats_OFFLINE_0
m

Mayank

04/11/2022, 2:35 PM
Are you sure vpn is disconnected? You should see localhost instead of ip address
l

Luis Fernandez

04/11/2022, 4:46 PM
yea it’s shut
unless this is not on by defcault ?
pinot.set.instance.id.to.hostname
again i’m trying to run the examples from the IDE.
in this case i’m running the QuickStart batch example
m

Mayank

04/11/2022, 6:24 PM
Hmm quickstart should run as-is, without any changes. Do you have a router that might be in play here?
l

Luis Fernandez

04/11/2022, 6:34 PM
yes i’m behind a router
m

Mayank

04/11/2022, 6:38 PM
Just to debug, can you disconnect from router and try?
l

Luis Fernandez

04/11/2022, 6:39 PM
it’s still saying the same when i turn my wifi off
2022/04/11 14:39:03.485 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1
like host keeps on being those numbers
m

Mayank

04/11/2022, 6:40 PM
@Tim Santos ^^
n

Navina

04/11/2022, 7:08 PM
@Luis Fernandez let me take a stab at unpuzzling this 🙂 1. what does
ifconfig
on your machine return? 2. Are you passing any explicit
brokerHost
in your quickstart command line args? If not, you can try to debug with a breakpoint in class
NetUtils.getHostAddress
l

Luis Fernandez

04/11/2022, 7:09 PM
nah i’m not sending anything
n

Navina

04/11/2022, 7:10 PM
ok can you try #1 and #2 ? also, are you running on a linux or mac machine?
l

Luis Fernandez

04/11/2022, 7:10 PM
mac machine
Copy code
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en3: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether ac:de:48:00:11:22 
	inet6 fe80::aede:48ff:fe00:1122%en3 prefixlen 64 scopeid 0x4 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (100baseTX <full-duplex>)
	status: active
ap1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether a6:83:e7:83:1b:87 
	media: autoselect
	status: inactive
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether a4:83:e7:83:1b:87 
	inet6 fe80::cf0:694d:2fc1:10da%en0 prefixlen 64 secured scopeid 0x6 
	inet 192.168.1.239 netmask 0xffffff00 broadcast 192.168.1.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:c7:97:27:dc:00 
	media: autoselect <full-duplex>
	status: inactive
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:c7:97:27:dc:01 
	media: autoselect <full-duplex>
	status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
	ether 82:c7:97:27:dc:01 
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: en1 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 8 priority 0 path cost 0
	member: en2 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 7 priority 0 path cost 0
	nd6 options=201<PERFORMNUD,DAD>
	media: <unknown type>
	status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether d2:cc:1b:7e:45:87 
	inet6 fe80::d0cc:1bff:fe7e:4587%awdl0 prefixlen 64 scopeid 0xa 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
llw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether d2:cc:1b:7e:45:87 
	inet6 fe80::d0cc:1bff:fe7e:4587%llw0 prefixlen 64 scopeid 0xb 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::1436:ab02:fa43:f0eb%utun0 prefixlen 64 scopeid 0xc 
	nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
	inet6 fe80::301f:7bae:f0a2:3cab%utun1 prefixlen 64 scopeid 0xd 
	nd6 options=201<PERFORMNUD,DAD>
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
	inet6 fe80::ce81:b1c:bd2c:69e%utun2 prefixlen 64 scopeid 0xe 
	nd6 options=201<PERFORMNUD,DAD>
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6467<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether 00:e0:4c:00:16:7e 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (none)
	status: inactive
if config
message has been deleted
n

Navina

04/11/2022, 7:18 PM
can't say I fully understand the ifconfig output. but looks fine for the most part of it.
So you can walk through the debugger to see in which part of the jdk code does it make the system call for
hostname
and see what it returns. another option is to maybe try providing input arguments for
brokerHost
and
brokerPort
.
@Luis Fernandez do try the above ☝️ and let us know. You can also check these commands on your terminal -
hostname
or
host 0.0.26.108
to see what is going on because java uses whatever is configured in your system
also, iirc, you can check
cat /etc/hosts
for any custom overrides
l

Luis Fernandez

04/11/2022, 7:56 PM
Copy code
$ host 0.0.26.108
Host 108.26.0.0.in-addr.arpa. not found: 3(NXDOMAIN)
do i run that while running the code?
Copy code
127.0.0.1	localhost
::1             localhost
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
Copy code
$ hostname
6764
n

Navina

04/11/2022, 8:50 PM
this ☝️ looks weird. why is your hostname a number? are you a on VM of some kind or bare metal ?
l

Luis Fernandez

04/11/2022, 8:51 PM
oh this is cause I'm my work laptop which just has an id as the name I would assume
n

Navina

04/11/2022, 8:53 PM
so if you do
host 6764
does it resolve correctly?
wait my bad. that won't be the host command.
l

Luis Fernandez

04/11/2022, 9:02 PM
what do you recommend to run
n

Navina

04/11/2022, 9:11 PM
since your /etc/hosts doesn't have any override, I would assume something in the lookup is maybe going awry. have you tried
dig localhost
?
l

Luis Fernandez

04/11/2022, 9:12 PM
Copy code
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;localhost.			IN	A

;; ANSWER SECTION:
localhost.		10800	IN	A	127.0.0.1
n

Navina

04/11/2022, 9:18 PM
yikes. weird. so I am able to reproduce it with a debugger when I set
InetAddress#getLocalHost
to return
6764
. I do see the output resolution that you see, which is
0.0.26.108
Unless the hostname returned in
localhost
, it doesn't look at the loopback address
127.0.0.1
in your case.
@Luis Fernandez if you are blocked on this, I would recommend trying the quickstart with explicit
-brokerHost
program argument with value as
127.0.01
, while we try to figure out this network resolution behavior
l

Luis Fernandez

04/11/2022, 9:31 PM
noob question intellij has changed a bit if i want to send arguments
it would be like this no?
-dataDir "usr/local/var/lib/pinot/data" -brokerHost "127.0.01"
message has been deleted
n

Navina

04/11/2022, 9:32 PM
yep that's right. there is a typo in your ip address
l

Luis Fernandez

04/11/2022, 9:33 PM
Copy code
picocli.CommandLine$UnmatchedArgumentException: Unknown options: '-brokerHost', '127.0.0.1'
	at picocli.CommandLine$Interpreter.validateConstraints(CommandLine.java:13143) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13095) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.processSubcommand(CommandLine.java:13343) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.processArguments(CommandLine.java:13260) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13072) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13041) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:12942) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine.parseArgs(CommandLine.java:1478) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:149) [classes/:?]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192) [classes/:?]
	at org.apache.pinot.tools.Quickstart.main(Quickstart.java:229) [classes/:?]
n

Navina

04/11/2022, 9:38 PM
wait. i thought you were running some batch quickstart. Let me check this command
l

Luis Fernandez

04/11/2022, 9:42 PM
that’s on
QuickStart.java
it dies when it reaches here
and that’s the output
well it doesn’t die hehe it just throws that error we have been looking at
n

Navina

04/12/2022, 4:25 AM
Ok. I think you have more than one active network interface and hence, the
InetAddress.getLocalhost
is probably not returning the ip that is accessible externally. Going by this - https://stackoverflow.com/questions/9481865/getting-the-ip-address-of-the-current-machine-using-java Perhaps we just need a way to give you a way to override the brokerHost for that command? Also, have you tried other quick start commands in your IDE? Do they all have the same problem?
🙌 1
l

Luis Fernandez

04/12/2022, 2:08 PM
i have ran 2 so far and both have the same issue
maybe there’s a better way to do this but i just did this:
message has been deleted
in
NetUtils.getHostAddress
just to do my thing in my local env
m

Mayank

04/12/2022, 2:13 PM
This is definitely due to some on standard setup at your end. But since it is for debugging only, hard coding works I guess.
l

Luis Fernandez

04/12/2022, 2:15 PM
thank you both for all the help couldn’t have gone around it without it
👍 1
m

Mayank

04/12/2022, 2:27 PM
Thanks for your persistence