https://pinot.apache.org/ logo
Join Slack
Powered by
<https://github.com/cyrilou242/thirdeye-oss> Is it...
# thirdeye-pinot
l
https://github.com/cyrilou242/thirdeye-oss Is it separated from Pinot independently?
c
yes this is a fork that is used in production in my previous company. it only contains the ThirdEye code.
separating ThirdEye and Pinot was a way to keep focus in both projects
l
I got it. I'm struggling to install mysql 5.7 now.
c
the quickstart I mentioned here: https://github.com/cyrilou242/thirdeye-oss/tree/master/quickstart contains a dockerized mysql 5
l
If I use docker, isn't there any part to be customized?
c
not sure to understand the question
l
I think that I'll need to change table in the database. To use my custom data.
I mean, after pulling and running this docker, can I edit custom dataset on frontend?
c
Yes. Not in the frontend but in a config file. The dockerized quickstart mounts a local volume to make it easy to update the config files. So it’s easy to update the config for your data source. I suggest the following: • Go through the dockerized quickstart. Ensure everything is working well. • Edit the data-sources config in the config folder to point to your Pinot instance. Restart the dockerized TE.
l
Could you show me an example how to set pinot dataset in config file? just mock data in the example.
c
l
yes, here how can I get zookeeperurl for ex?
c
how is your Pinot installed?
l
I've installed Pinot and already set dataset there
c
how did you install Pinot? docker, local, k8s?
l
using docker
Copy code
version: '3.7'
services:
  zookeeper:
    image: zookeeper:3.5.6
    hostname: zookeeper
    container_name: manual-zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  pinot-controller:
    image: apachepinot/pinot:0.9.3
    command: "StartController -zkAddress manual-zookeeper:2181"
    container_name: "manual-pinot-controller"
    volumes:
      - /home/csguru/workspace/Sapient/works/pinot:/tmp/pinot-quick-start    
    restart: unless-stopped
    ports:
      - "9000:9000"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
    depends_on:
      - zookeeper
  pinot-broker:
    image: apachepinot/pinot:0.9.3
    command: "StartBroker -zkAddress manual-zookeeper:2181"
    restart: unless-stopped
    container_name: "manual-pinot-broker"
    ports:
      - "8099:8099"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
    depends_on:
      - pinot-controller
  pinot-server:
    image: apachepinot/pinot:0.9.3
    command: "StartServer -zkAddress manual-zookeeper:2181"
    restart: unless-stopped
    container_name: "manual-pinot-server" 
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
    depends_on:
      - pinot-broker
c
ok so zookeeper is on port 2181
l
then, zookeeperurl will be localhost:2181 ?
c
yes
l
I see.
I'll try all after docker pulling and will get back to you if I have any question. Thanks so much.
c
Copy code
dataSourceConfigs:
  - className: org.apache.pinot.thirdeye.datasource.pinot.PinotThirdEyeDataSource
    properties:
        zookeeperUrl: 'localhost:2181'
        clusterName: 'PinotCluster'
        controllerConnectionScheme: 'http'
        controllerHost: '127.0.0.1'
        controllerPort: 9000
        cacheLoaderClassName: org.apache.pinot.thirdeye.datasource.pinot.PinotD2ResponseCacheLoader
    metadataSourceConfigs:
      - className: org.apache.pinot.thirdeye.auto.onboard.AutoOnboardPinotMetadataSource
should work
l
Ok, I'll try that.
Copy code
Step 8/20 : RUN curl -o- <https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh> | bash
 && export NVM_DIR="$HOME/.nvm"    && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"    && nvm
install v10.24.1    && npm install -g phantomjs --unsafe-perm --ignore-scripts
 ---> Running in a57163dfcfda
 % Total   % Received % Xferd Average Speed  Time   Time    Time Current
                                Dload Upload  Total  Spent   Left Speed
 0    0   0    0   0    0     0     0 --:--:-- 0:00:05 --:--:--    0
curl: (7) Failed to connect to <http://raw.githubusercontent.com|raw.githubusercontent.com> port 443: Connection refused
When I build a docker, it gives this error.
There's an error when I ran this.
c
l
I can see that script in browser.
But can't run Dockerfile in building.
I don't understand why it happens
Is there a way to download install.sh at my local and go to next step?
c
let’s try to understand the problem first
l
Copy code
Step 8/20 : RUN curl -o- <https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh> | bash    && export NVM_DIR="$HOME/.nvm"
    && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"    && nvm install v10.24.1    && npm install -g phantomjs --unsafe-perm
 --ignore-scripts 
 ---> Running in 07a284672750 
 % Total   % Received % Xferd Average Speed  Time   Time    Time Current 
                                Dload Upload  Total  Spent   Left Speed                                                     
 0    0   0    0   0    0     0     0 --:--:-- 0:00:05 --:--:--    0                                                    
curl: (7) Failed to connect to <http://raw.githubusercontent.com|raw.githubusercontent.com> port 443: Connection refused                                             
The command '/bin/sh -c curl -o- <https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh> | bash    && export NVM_DIR="$HO
ME/.nvm"    && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"    && nvm install v10.24.1    && npm install -g phantomjs --uns
afe-perm --ignore-scripts' returned a non-zero code: 1
This is what I'm getting now.
I tried this operation on Cloud machine and it works fine, but it gave me error in next step.
Copy code
# run 
docker-compose up front workers
c
in local: problem seems related to your machine configuration/network access . Try to understand the problem. You may want to search on google. in the cloud machine: what is the error?
l
This is a full log on the cloud machine.
error.log
c
thirdeye cannot connecto the mysql db. I guess you did not launch the mysql db in the cloud machine
l
This is docker starting, so it is automatically connect to mysql db. Isn't it?
I ran mysql db like this
Copy code
docker-compose up -d mysql
c
I the tutorial:
Copy code
docker-compose up -d mysql
export PERSISTENCE_MYSQL_PORT=3307
export DATASOURCE_MYSQL_PORT=3307

make create-persistence-db-tables

# creation in database testdata - replace by your own when you're ready
make create-datasource-tables
l
Yeah, I did it all.
c
can you run:
Copy code
cat config/data-sources/data-sources-config.yml
l
Copy code
dataSourceConfigs: 
 - className: org.apache.pinot.thirdeye.datasource.sql.SqlThirdEyeDataSource 
   properties: 
     MySQL: 
       - db: 
           ## notice the &useSSL=false 
           dataset: jdbc:<mysql://172.17.0.1:3307/testdata?autoReconnect=true&useSSL=false> 
         user: root 
         password: "" 
## add other sql datasources here if you want 
## just hardcode things for test 
#       - other_db: 
#           dataset: jdbc:<mysql://mySqlDBHost:3307/testdata> 
#         user: haha 
#         password: blabla
c
echo $PERSISTENCE_MYSQL_PORT
l
3307
c
Copy code
mysql -u root -p
then hit enter
l
ERROR 1698 (28000): Access denied for user 'root'@'localhost'
c
oh should not work actually
l
with sudo, I can login mysql
c
it’s not related to sudo
l
Copy code
datasci@alert-system:~/thirdeye-oss/quickstart$ sudo mysql -u root 
Welcome to the MySQL monitor. Commands end with ; or \g. 
Your MySQL connection id is 9 
Server version: 8.0.29-0ubuntu0.20.04.3 (Ubuntu) 
 
Copyright (c) 2000, 2022, Oracle and/or its affiliates. 
 
Oracle is a registered trademark of Oracle Corporation and/or its 
affiliates. Other names may be trademarks of their respective 
owners. 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. 
 
mysql>
c
it’s not the same instance
instance lives on 3307
l
hmm
c
Copy code
mysql --host localhost:3307 -u root -p
something like this
l
ERROR 2005 (HY000): Unknown MySQL server host 'localhost:3307' (-2)
c
I don’t know the mysql cli by heart. I think port is another parameter
You should try: localhost + port 3307 172.17.0.1 + port 3307 127.0.0.1 + port 3307
l
mysql -h localhost -u root -p Enter password: ERROR 1698 (28000): Access denied for user 'root'@'localhost'
c
this does not pass the port
Copy code
mysql -h 127.0.0.1 -P 3307 -u root -p
mysql -h localhost -P 3307 -u root -p
mysql -h 172.17.0.1 -P 3307 -u root -p
l
Copy code
mysql -h 127.0.0.1 -P 3307 -u root -p
It's working
c
oh cool
then in
Copy code
config/data-sources/data-sources-config.yml
can you replace the mysql url by the one that is working
the localhost url behavior of docker actually depends on the kind of linux
l
Copy code
dataset: jdbc:<mysql://127.0.0.1:3307/testdata?autoReconnect=true&useSSL=false>
It this okay?
c
yes - also change it in
Copy code
config/persistence.yml
l
Copy code
databaseConfiguration: 
 ## notice useSSL=false 
 url: jdbc:<mysql://127.0.1:3307/thirdeye?autoReconnect=true&useSSL=false>                    
 user: root 
 password: "" 
 driver: com.mysql.jdbc.Driver
c
yes
l
now building docker again?
c
no need to rebuild you can relaunch
Copy code
docker-compose up front workers
l
error_1.log
error_1.log
c
ok sorry messed up. 127.0.0.1 will not work from a docker container. The idea is that you need to hit the mysql on localhost:3307 from a docker container the url depends on your system. you can have a look at: https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach
l
So is it okay to use public IP instead of 127.0..0.1?
c
no neither
it will be something like
Copy code
host.docker.internal
172.17.42.1
oh actually you can try:
Copy code
Use --network="host" in your docker run command, then 127.0.0.1 in your docker container will point to your docker host.

Note: This mode only works on Docker for Linux, per the documentation
l
I didn't get you. Where do I change this 
--network="host"
c
you can also try to connect to mysql from a docker image and find the good port:
Copy code
docker run -it mysql:5.7 /bin/sh

# then try
mysql -h 127.0.0.1 -P 3307 -u root -p
mysql -h localhost -P 3307 -u root -p
mysql -h 172.17.0.1 -P 3307 -u root -p
mysql -h 172.17.42.1 -P 3307 -u root -p
l
mysql -h 172.17.0.1 -P 3307 -u root -p
This works
c
ok then set it in the two files
l
Done.
It seems ThirdEye is running now.
But it's empty data.
c
yes - please read the following of the quickstart to connecto the demo data if you’re confident you can also try to connect your data source directly as you prefer
l
Okay. let me try with that.
Is it possible to use Pinot dataset in data-source-config file like this?
@Cyril de Catheu one question for now. We set datasource as mysql db and I think it's for thirdeye data storing. Now I need to ingest data from pinot cluster. To do that, I have to replace data-source-config with ponit-cluster or add new className?
When I use Pinot-cluster datasource, got an error.
Copy code
front_1    | {"exception":"java.net.ConnectException: Connection refused\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)\n\tat org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)\n\tat org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)\n","level":"WARN","logger":"org.apache.zookeeper.ClientCnxn","thread":"pool-6-thread-18-SendThread(localhost:2181)","message":"Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect","timestamp":1653328100865}
c
The config in the screenshot is almost correct just replace 127.0.0.1 and localhost by 172.17.0.1
2 Views
Open in Slack
PreviousNext
Powered by