Hello Setting up pinot on a docker compose container Everthi

Question

Hello Setting up pinot on a docker compose container Everthings runs as expected but when try to create a stream table by using the below commands getting Connection refused error ```bin pinot admin sh AddTable schemaFile examples stream airlineStats airlineStats schema json tableConfigFile examples stream airlineStats airlineStats realtime table config json exec``` Error ``` Got Exception to upload Pinot Schema airlineStats shaded org apache http conn HttpHostConnectException Connect to 172 26 0 5 9000 172 26 0 5 failed Connection refused Connection refused at shaded org apache http impl conn DefaultHttpClientConnectionOperator connect DefaultHttpClientConnectionOperator java 156 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl conn PoolingHttpClientConnectionManager connect PoolingHttpClientConnectionManager java 374 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain MainClientExec establishRoute MainClientExec java 393 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain MainClientExec execute MainClientExec java 236 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain ProtocolExec execute ProtocolExec java 186 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain RetryExec execute RetryExec java 89 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain RedirectExec execute RedirectExec java 110 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl client InternalHttpClient doExecute InternalHttpClient java 185 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl client CloseableHttpClient execute CloseableHttpClient java 83 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl client CloseableHttpClient execute CloseableHttpClient java 108 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot common utils FileUploadDownloadClient sendRequest FileUploadDownloadClient java 383 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot common utils FileUploadDownloadClient addSchema FileUploadDownloadClient java 448 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin command AddTableCommand uploadSchema AddTableCommand java 135 pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin command AddTableCommand execute AddTableCommand java 170 pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin PinotAdministrator execute PinotAdministrator java 164 pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin PinotAdministrator main PinotAdministrator java 184 pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed Caused by java net ConnectException Connection refused Connection refused at java net PlainSocketImpl socketConnect Native Method ~ 1 8 0 282 at java net AbstractPlainSocketImpl doConnect AbstractPlainSocketImpl java 350 ~ 1 8 0 282 at java net AbstractPlainSocketImpl connectToAddress AbstractPlainSocketImpl java 206 ~ 1 8 0 282 at java net AbstractPlainSocketImpl connect AbstractPlainSocketImpl java 188 ~ 1 8 0 282 at java net SocksSocketImpl connect SocksSocketImpl java 392 ~ 1 8 0 282 at java net Socket connect Socket java 607 ~ 1 8 0 282 at shaded org apache http conn socket PlainConnectionSocketFactory connectSocket PlainConnectionSocketFactory java 75 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl conn DefaultHttpClientConnectionOperator connect DefaultHttpClientConnectionOperator java 142 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed 15 more Exception caught shaded org apache http conn HttpHostConnectException Connect to 172 26 0 5 9000 172 26 0 5 failed Connection refused Connection refused at shaded org apache http impl conn DefaultHttpClientConnectionOperator connect DefaultHttpClientConnectionOperator java 156 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl conn PoolingHttpClientConnectionManager connect PoolingHttpClientConnectionManager java 374 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain MainClientExec establishRoute MainClientExec java 393 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain MainClientExec execute MainClientExec java 236 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain ProtocolExec execute ProtocolExec java 186 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain RetryExec execute RetryExec java 89 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl execchain RedirectExec execute RedirectExec java 110 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl client InternalHttpClient doExecute InternalHttpClient java 185 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl client CloseableHttpClient execute CloseableHttpClient java 83 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl client CloseableHttpClient execute CloseableHttpClient java 108 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot common utils FileUploadDownloadClient sendRequest FileUploadDownloadClient java 383 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot common utils FileUploadDownloadClient addSchema FileUploadDownloadClient java 448 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin command AddTableCommand uploadSchema AddTableCommand java 135 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin command AddTableCommand execute AddTableCommand java 170 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin PinotAdministrator execute PinotAdministrator java 164 pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at org apache pinot tools admin PinotAdministrator main PinotAdministrator java 184 pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed Caused by java net ConnectException Connection refused Connection refused at java net PlainSocketImpl socketConnect Native Method ~ 1 8 0 282 at java net AbstractPlainSocketImpl doConnect AbstractPlainSocketImpl java 350 ~ 1 8 0 282 at java net AbstractPlainSocketImpl connectToAddress AbstractPlainSocketImpl java 206 ~ 1 8 0 282 at java net AbstractPlainSocketImpl connect AbstractPlainSocketImpl java 188 ~ 1 8 0 282 at java net SocksSocketImpl connect SocksSocketImpl java 392 ~ 1 8 0 282 at java net Socket connect Socket java 607 ~ 1 8 0 282 at shaded org apache http conn socket PlainConnectionSocketFactory connectSocket PlainConnectionSocketFactory java 75 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed at shaded org apache http impl conn DefaultHttpClientConnectionOperator connect DefaultHttpClientConnectionOperator java 142 ~ pinot all 0 7 1 jar with dependencies jar 0 7 1 afa4b252ab1c424ddd6c859bb305b2aa342b66ed 15 more``` ```DOCKER HOST=unix var run docker sock docker ps ``` ```CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 545bf572fd0d apachepinot pinot release 0 7 1 bin pinot admin s 2 hours ago Up 2 hours 8096 8097 tcp 8099 tcp 9000 tcp 0 0 0 0 8098 gt 8098 tcp 8098 gt 8098 tcp composetest pinot server 1 4033365f967e apachepinot pinot release 0 7 1 bin pinot admin s 2 hours ago Up 2 hours 8096 8098 tcp 9000 tcp 0 0 0 0 8099 gt 8099 tcp 8099 gt 8099 tcp composetest pinot broker 1 1712fba17460 apachepinot pinot release 0 7 1 bin pinot admin s 2 hours ago Up 2 hours 8096 8099 tcp 0 0 0 0 9000 gt 9000 tcp 9000 gt 9000 tcp composetest pinot controller 1 aeac35e847b6 confluentinc cp kafka 5 3 0 etc confluent dock 2 days ago Up 2 hours 0 0 0 0 9092 gt 9092 tcp 9092 gt 9092 tcp composetest kafka 1 050bf6511ab9 zookeeper latest docker entrypoint 2 days ago Up 2 hours 2888 tcp 3888 tcp 0 0 0 0 2181 gt 2181 tcp 2181 gt 2181 tcp 8080 tcp composetest zookeeper 1``` Can anyone kindly suggest how to proceed from here

Xiang Fu · Answer

where are you running this addTable command

Xiang Fu · Answer

you need to run it inside pinot controller container

Xiang Fu · Answer

or ensure you can access pinot controller 9000 from where you want to issue the request

Ravi R · Answer

< Xiang Fu> Thanks I tried it from pinot controller and got this error now Even when I try to do ingest data via swagger API getting the same error ` code 500 error org apache kafka common errors TimeoutException Timeout expired while fetching topic metadata `

Xiang Fu · Answer

seems the issue is that your kafka doesn t open to pinot container

Ravi R · Answer

```FYI have created a kafka topic aleady My docker compose yml file is version 3 services zookeeper image zookeeper latest hostname zookeeper ports 2181 2181 environment ZOO MY ID 1 ZOO PORT 2181 ZOO SERVERS server 1=zookeeper 2888 3888 2181 volumes pinot docker demo zookeeper data data pinot docker demo zookeeper datalog datalog kafka image confluentinc cp kafka 5 3 0 hostname kafka ports 9092 9092 environment KAFKA ADVERTISED LISTENERS LISTENER DOCKER LISTENER DOCKER KAFKA LISTENER SECURITY PROTOCOL MAP LISTENER DOCKER INTERNAL PLAINTEXT LISTENER DOCKER EXTERNAL PLAINTEXT KAFKA INTER BROKER LISTENER NAME LISTENER DOCKER INTERNAL KAFKA ZOOKEEPER CONNECT zookeeper 2181 kafka KAFKA BROKER ID 1 KAFKA LOG4J LOGGERS kafka controller=INFO kafka producer async DefaultEventHandler=INFO state change logger=INFO KAFKA OFFSETS TOPIC REPLICATION FACTOR 1 volumes pinot docker demo kafka data var lib kafka data depends on zookeeper pinot controller image apachepinot pinot release 0 7 1 hostname pinot controller volumes pinot docker demo pinot controller tmp data controller ports 9000 9000 command StartController zkAddress zookeeper 2181 depends on kafka pinot broker image apachepinot pinot release 0 7 1 hostname pinot broker ports 8099 8099 command StartBroker zkAddress zookeeper 2181 depends on zookeeper kafka pinot controller pinot server image apachepinot pinot release 0 7 1 hostname pinot server volumes pinot docker demo pinot server tmp data server ports 8098 8098 command StartServer zkAddress zookeeper 2181 depends on zookeeper kafka pinot controller```

Ravi R · Answer

< Xiang Fu> Possible So how do I enable it Also Do you see anything missing in my docker compose file

Xiang Fu · Answer

what s your create table config

Xiang Fu · Answer

you need to change the kafka broker to `kafka 9092`

Ravi R · Answer

Already logged in on docker but still getting this issue `docker Error response from daemon pull access denied for startcontroller repository does not exist or may require docker login denied requested access to the resource is denied ` Any suggestion < Xiang Fu> < Neha Pawar>

Xiang Fu · Answer

whats the cmd are you using

Xiang Fu · Answer

i think the kafka container name is `composetest kafka 1`

Ravi R · Answer

< Xiang Fu> To start the pinot controller running ```docker run rm ti network=pinot demo name pinot controller p 9000 9000 e JAVA OPTS= Dplugins dir= opt pinot plugins Xms1G Xmx4G XX +UseG1GC XX MaxGCPauseMillis=200 XX +PrintGCDetails XX +PrintGCDateStamps XX +PrintGCApplicationStoppedTime XX +PrintGCApplicationConcurrentTime Xloggc gc pinot controller log d $ PINOT IMAGE StartController zkAddress pinot zookeeper 2181```

Ravi R · Answer

only change in the above command is that changed image name StartController to the lower case as it keep throwing me error `docker invalid reference format repository name must be lowercase `

Xiang Fu · Answer

```StartController``` is a pinot command so it has to be exact match