https://pinot.apache.org/ logo
Join Slack
Powered by
in the pinot docs we have this about <start pinot ...
# troubleshooting
l
in the pinot docs we have this about start pinot via IDE but how is this starting the pinot infra via the IDE? I guess that ultimately my question is how can I attach remote debugger to my local pinot processes
m
It is suggesting to start the 
quickStart
program, which internally starts all pinot components within the same jvm. You can run Pinot and debug it in IDE as you do any application.
l
i got this exception do yo uknow why it may be?
Copy code
Instance 0.0.26.108_9000 is not leader of cluster QuickStartCluster due to exception happen when session check
org.I0Itec.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException
i was trying to run the empty quick start model https://github.com/apache/pinot/pull/8024
m
This is a newer feature. @Kenny Bastani any idea on what might be going on?
k
@Luis Fernandez What command are you using to start Pinot?
l
like this sh pinot-admin.sh QuickStart -type EMPTY -dataDir “usr/local/var/lib/pinot/data”
k
Do you have ZK running externally?
l
no
but i also don’t have zookeeper running locally do i have to run zk manually first i thought this would start zk for me
k
Yes, it will
One sec
Please run this command
netstat -vanp tcp | grep '*.2123\|9000\|8000\|7000'
And let me know what the output is
Also, 
ls /usr/local/var/lib/pinot/data/rawdata
@Luis Fernandez Let me know if you got it solved. Happy to jump on a call if you need help with anything.
l
hey Kenny sorry, was out yesterday already, that netstat command is while i’m trying to run the command yes?
also i’m running this from the source code itself, so i’m running from 
pinot/build
don’t know if that has anything to do
this is the output of the first command you put
Copy code
tcp6       0      0  *.7000                 *.*                    LISTEN      131072 131072   1905      0 0x0100 0x00000006
tcp4       0      0  *.7000                 *.*                    LISTEN      131072 131072   1905      0 0x0100 0x00000006
oh i’m dumb hehe it was airplay receiver it’s on the pinot docs thank you both for your help i think we gucci
👍 1
@Mayank when i saw the title running from IDE I was thinking that it would literally start off from the IDE like run everything from intellij for example, do you know who may know how to setup debugging for this? I have it running but i’m unsure as to how to debug
m
Yes, you can literally start off from IDE. You can pick one of the quick start classes and hit the green run button in the main function from within IDE.
l
anyone familiar with this error
Copy code
2022/04/08 15:53:37.705 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1
java.net.NoRouteToHostException: No route to host (Host unreachable)
anyone familiar with this error while executing the QuickStart from IDE
Copy code
2022/04/08 15:53:37.705 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1
java.net.NoRouteToHostException: No route to host (Host unreachable)
m
Is broker up and running at port 8000 of 0.0.26.108?
l
when i execute queries i get that error
Copy code
2022/04/08 20:00:59.462 INFO [StartBrokerCommand] [main] Executing command: StartBroker -brokerHost null -brokerPort 8000 -zkAddress localhost:2123
Copy code
2022/04/08 20:01:00.457 INFO [StartServiceManagerCommand] [Start a Pinot [BROKER]] Started Pinot [BROKER] instance [Broker_0.0.26.108_8000] at 9.464s since launch
k
Why is your IP not localhost?
l
That's an amazing question 😂
m
Is it not the local host’s IP?
l
0.0.26 i guess it’s good
but i don’t understand what that error is about
is there anything in particular i have to do if i’m running in mac?
m
Is it M1 or are you on vpn?
l
that’s the processor yes?
I’m on vpn yes
i’m not on m1 but def have vpn on at the moment
m
Try with vpn disconnected
My guess is that it is messing up the connectivity
l
aw i still see the error
Copy code
java.net.NoRouteToHostException: No route to host (Host unreachable)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/java.net.Socket.connect(Socket.java:558)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1248)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1337)
	at org.apache.pinot.tools.admin.command.AbstractBaseAdminCommand.sendRequest(AbstractBaseAdminCommand.java:91)
	at org.apache.pinot.tools.admin.command.PostQueryCommand.run(PostQueryCommand.java:145)
	at org.apache.pinot.tools.admin.command.QuickstartRunner.runQuery(QuickstartRunner.java:280)
	at org.apache.pinot.tools.Quickstart.runSampleQueries(Quickstart.java:192)
	at org.apache.pinot.tools.Quickstart.execute(Quickstart.java:130)
	at org.apache.pinot.tools.admin.command.QuickStartCommand.execute(QuickStartCommand.java:157)
	at org.apache.pinot.tools.Command.call(Command.java:33)
	at org.apache.pinot.tools.Command.call(Command.java:29)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:161)
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192)
	at org.apache.pinot.tools.Quickstart.main(Quickstart.java:229)
there’s also this
Copy code
2022/04/11 10:29:11.620 ERROR [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread] Attempts exceeded when downloading segment: baseballStats_OFFLINE_0 for table: baseballStats_OFFLINE from: <http://0.0.26.108:9000/segments/baseballStats/baseballStats_OFFLINE_0> to: /Users/luisfernandez/development/pinot/usr/local/var/lib/pinot/data/1649687329197/baseballStats/rawdata/PinotServerDataDir0/baseballStats_OFFLINE/tmp/tmp-baseballStats_OFFLINE_0-edf388f0-9451-4948-8730-a6cf9987103e/baseballStats_OFFLINE_0.tar.gz
2022/04/11 10:29:11.624 ERROR [SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread] Caught exception in state transition from OFFLINE -> ONLINE for resource: baseballStats_OFFLINE, partition: baseballStats_OFFLINE_0
m
Are you sure vpn is disconnected? You should see localhost instead of ip address
l
yea it’s shut
unless this is not on by defcault ? 
pinot.set.instance.id.to.hostname
again i’m trying to run the examples from the IDE.
in this case i’m running the QuickStart batch example
m
Hmm quickstart should run as-is, without any changes. Do you have a router that might be in play here?
l
yes i’m behind a router
m
Just to debug, can you disconnect from router and try?
l
it’s still saying the same when i turn my wifi off
2022/04/11 14:39:03.485 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1
like host keeps on being those numbers
m
@Tim Santos ^^
n
@Luis Fernandez let me take a stab at unpuzzling this 🙂 1. what does 
ifconfig
on your machine return? 2. Are you passing any explicit 
brokerHost
in your quickstart command line args? If not, you can try to debug with a breakpoint in class 
NetUtils.getHostAddress
l
nah i’m not sending anything
n
ok can you try #1 and #2 ? also, are you running on a linux or mac machine?
l
mac machine
Copy code
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en3: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether ac:de:48:00:11:22 
	inet6 fe80::aede:48ff:fe00:1122%en3 prefixlen 64 scopeid 0x4 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (100baseTX <full-duplex>)
	status: active
ap1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether a6:83:e7:83:1b:87 
	media: autoselect
	status: inactive
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether a4:83:e7:83:1b:87 
	inet6 fe80::cf0:694d:2fc1:10da%en0 prefixlen 64 secured scopeid 0x6 
	inet 192.168.1.239 netmask 0xffffff00 broadcast 192.168.1.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:c7:97:27:dc:00 
	media: autoselect <full-duplex>
	status: inactive
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:c7:97:27:dc:01 
	media: autoselect <full-duplex>
	status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
	ether 82:c7:97:27:dc:01 
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: en1 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 8 priority 0 path cost 0
	member: en2 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 7 priority 0 path cost 0
	nd6 options=201<PERFORMNUD,DAD>
	media: <unknown type>
	status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether d2:cc:1b:7e:45:87 
	inet6 fe80::d0cc:1bff:fe7e:4587%awdl0 prefixlen 64 scopeid 0xa 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
llw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether d2:cc:1b:7e:45:87 
	inet6 fe80::d0cc:1bff:fe7e:4587%llw0 prefixlen 64 scopeid 0xb 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::1436:ab02:fa43:f0eb%utun0 prefixlen 64 scopeid 0xc 
	nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
	inet6 fe80::301f:7bae:f0a2:3cab%utun1 prefixlen 64 scopeid 0xd 
	nd6 options=201<PERFORMNUD,DAD>
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
	inet6 fe80::ce81:b1c:bd2c:69e%utun2 prefixlen 64 scopeid 0xe 
	nd6 options=201<PERFORMNUD,DAD>
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6467<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether 00:e0:4c:00:16:7e 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (none)
	status: inactive
if config
message has been deleted
n
can't say I fully understand the ifconfig output. but looks fine for the most part of it.
So you can walk through the debugger to see in which part of the jdk code does it make the system call for 
hostname
and see what it returns. another option is to maybe try providing input arguments for 
brokerHost
and 
brokerPort
.
@Luis Fernandez do try the above ☝️ and let us know. You can also check these commands on your terminal - 
hostname
or 
host 0.0.26.108
to see what is going on because java uses whatever is configured in your system
also, iirc, you can check 
cat /etc/hosts
for any custom overrides
l
Copy code
$ host 0.0.26.108
Host 108.26.0.0.in-addr.arpa. not found: 3(NXDOMAIN)
do i run that while running the code?
Copy code
127.0.0.1	localhost
::1             localhost
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
Copy code
$ hostname
6764
n
this ☝️ looks weird. why is your hostname a number? are you a on VM of some kind or bare metal ?
l
oh this is cause I'm my work laptop which just has an id as the name I would assume
n
so if you do 
host 6764
does it resolve correctly?
wait my bad. that won't be the host command.
l
what do you recommend to run
n
since your /etc/hosts doesn't have any override, I would assume something in the lookup is maybe going awry. have you tried 
dig localhost
?
l
Copy code
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;localhost.			IN	A

;; ANSWER SECTION:
localhost.		10800	IN	A	127.0.0.1
n
yikes. weird. so I am able to reproduce it with a debugger when I set 
InetAddress#getLocalHost
to return 
6764
. I do see the output resolution that you see, which is 
0.0.26.108
Unless the hostname returned in 
localhost
, it doesn't look at the loopback address 
127.0.0.1
in your case.
@Luis Fernandez if you are blocked on this, I would recommend trying the quickstart with explicit 
-brokerHost
program argument with value as 
127.0.01
, while we try to figure out this network resolution behavior
l
noob question intellij has changed a bit if i want to send arguments
it would be like this no?
-dataDir "usr/local/var/lib/pinot/data" -brokerHost "127.0.01"
message has been deleted
n
yep that's right. there is a typo in your ip address
l
Copy code
picocli.CommandLine$UnmatchedArgumentException: Unknown options: '-brokerHost', '127.0.0.1'
	at picocli.CommandLine$Interpreter.validateConstraints(CommandLine.java:13143) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13095) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.processSubcommand(CommandLine.java:13343) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.processArguments(CommandLine.java:13260) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13072) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13041) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:12942) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at picocli.CommandLine.parseArgs(CommandLine.java:1478) ~[picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:149) [classes/:?]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192) [classes/:?]
	at org.apache.pinot.tools.Quickstart.main(Quickstart.java:229) [classes/:?]
n
wait. i thought you were running some batch quickstart. Let me check this command
l
that’s on 
QuickStart.java
it dies when it reaches here
and that’s the output
well it doesn’t die hehe it just throws that error we have been looking at
n
Ok. I think you have more than one active network interface and hence, the 
InetAddress.getLocalhost
is probably not returning the ip that is accessible externally. Going by this - https://stackoverflow.com/questions/9481865/getting-the-ip-address-of-the-current-machine-using-java Perhaps we just need a way to give you a way to override the brokerHost for that command? Also, have you tried other quick start commands in your IDE? Do they all have the same problem?
🙌 1
l
i have ran 2 so far and both have the same issue
maybe there’s a better way to do this but i just did this:
message has been deleted
in 
NetUtils.getHostAddress
just to do my thing in my local env
m
This is definitely due to some on standard setup at your end. But since it is for debugging only, hard coding works I guess.
l
thank you both for all the help couldn’t have gone around it without it
👍 1
m
Thanks for your persistence
Open in Slack
PreviousNext
Powered by