https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
j
is this one the correct? 
ZOO_LOG4J_PROP: INFO,ROLLINGFILE
?
r
One unrelated question regarding the zookeeper logs - from the ls output it appears that the zk log files are not getting purged even though you've set
Copy code
ZK_AUTOPURGE_SNAPRETAINCOUNT: '3',
ZK_AUTOPURGE_PURGEINTERVAL: '1'
which means that only last 3 snapshots and logs should be present (ref : https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html#sc_strengthsAndLimitations). I wonder if these are the correct configs to use for enabling auto purge?
j
I am also setting 
ZK_ROLLING_LOG_FILE_MAX_COUNT: '10'
yeah I also thought above settings would purge logs but not sure if thats the correct setting or not
r
I'm not a k8s expert but if you're using zk docker image in k8s, then maybe you can check https://hub.docker.com/_/zookeeper
they seem to be mentioning 
ZOO_AUTOPURGE_PURGEINTERVAL
and 
ZOO_AUTOPURGE_SNAPRETAINCOUNT
j
I am setting those env vars by using helm values yaml available in Druid GitHub repo
r
Oh, I see. do you have github link to those vars? Sorry, I wasn't able to find them
j
Well you won't find the exact env vars in the file, but you know they exist
d
Zookeeper has a setting that auto prune consensus logs every hour, just turn that on.
j
what setting is that? are not they the one mentioned above?
d
In my Bitnami Zookeeper Helm chart, there’s this setting:
Copy code
autopurge:
  ## @param autopurge.snapRetainCount Retains the snapRetainCount most recent snapshots and the corresponding transaction logs and deletes the rest
  ##
  snapRetainCount: 3
  ## @param autopurge.purgeInterval The time interval in hours for which the purge task has to be triggered
  ## Set to a positive integer (1 and above) to enable the auto purging
  ##
  purgeInterval: 1
Your helm chart should have something similar.
j
Yeah exactly those env vars are the one mentioned above
And I have those in the statefulset for ZK
r
you could probably also checkout https://github.com/kubernetes-retired/contrib/blob/89f6948e24578fed2a90a87871b2263729f90ac3/statefulsets/zookeeper/README.md it mentions the properties as 
ZK_SNAP_RETAIN_COUNT
and 
ZK_PURGE_INTERVAL
I found this while searching for zk image using the properties you've mentioned
j
I am not sure what the actual env var is, if 
ZK_AUTOPURGE_SNAPRETAINCOUNT
or 
ZK_SNAP_RETAIN_COUNT
r
maybe you could add the other two env vars in the config too if possible. if they are right ones the logs should clear up. I think having extra env vars shouldn't cause any problems
j
this is what my ZK config look like now:
Copy code
zookeeper: {
                    enabled: true,
                    env: {
                        ZK_HEAP_SIZE: '512m',
                        ZK_INIT_LIMIT: '5',
                        ZK_SYNC_LIMIT: '2',
                        ZK_TICK_TIME: '2000',
                        ZK_MAX_CLIENT_CNXNS: '0',
                        ZK_AUTOPURGE_SNAPRETAINCOUNT: '3',
                        ZK_SNAP_RETAIN_COUNT: '3',
                        ZK_AUTOPURGE_PURGEINTERVAL: '1',
                        ZK_PURGE_INTERVAL: '1',
                        ZOO_LOG4J_PROP: 'INFO,ROLLINGFILE',
                        ZK_ROLLING_LOG_FILE_MAX_COUNT: '10'
                    },
                    persistence: {
                        size: '15Gi'
                    }
                }
btw, if I run an export in the pod I see the following
Copy code
zookeeper@druid-test-zookeeper-0:/apache-zookeeper-3.5.5-bin/bin$ export
declare -x AWS_DEFAULT_REGION="us-east-1"
declare -x AWS_REGION="us-east-1"
declare -x AWS_ROLE_ARN="arn:aws:iam::204717343847:role/druid-test-5533abc"
declare -x AWS_WEB_IDENTITY_TOKEN_FILE="/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
declare -x DRUID_CLUSTER_US2_BROKER_PORT="<tcp://172.20.107.228:8082>"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP="<tcp://172.20.107.228:8082>"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP_ADDR="172.20.107.228"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP_PORT="8082"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_BROKER_SERVICE_HOST="172.20.107.228"
declare -x DRUID_CLUSTER_US2_BROKER_SERVICE_PORT="8082"
declare -x DRUID_CLUSTER_US2_BROKER_SERVICE_PORT_HTTP="8082"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT="<tcp://172.20.186.60:8081>"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP="<tcp://172.20.186.60:8081>"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP_ADDR="172.20.186.60"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP_PORT="8081"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_COORDINATOR_SERVICE_HOST="172.20.186.60"
declare -x DRUID_CLUSTER_US2_COORDINATOR_SERVICE_PORT="8081"
declare -x DRUID_CLUSTER_US2_COORDINATOR_SERVICE_PORT_HTTP="8081"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT="<tcp://172.20.242.232:8083>"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP="<tcp://172.20.242.232:8083>"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP_ADDR="172.20.242.232"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP_PORT="8083"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_HISTORICAL_SERVICE_HOST="172.20.242.232"
declare -x DRUID_CLUSTER_US2_HISTORICAL_SERVICE_PORT="8083"
declare -x DRUID_CLUSTER_US2_HISTORICAL_SERVICE_PORT_HTTP="8083"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT="<tcp://172.20.38.117:8091>"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP="<tcp://172.20.38.117:8091>"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP_ADDR="172.20.38.117"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP_PORT="8091"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_SERVICE_HOST="172.20.38.117"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_SERVICE_PORT="8091"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_SERVICE_PORT_HTTP="8091"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT="<tcp://172.20.233.94:8888>"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP="<tcp://172.20.233.94:8888>"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP_ADDR="172.20.233.94"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP_PORT="8888"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_ROUTER_SERVICE_HOST="172.20.233.94"
declare -x DRUID_CLUSTER_US2_ROUTER_SERVICE_PORT="8888"
declare -x DRUID_CLUSTER_US2_ROUTER_SERVICE_PORT_HTTP="8888"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT="<tcp://172.20.192.66:2181>"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP="<tcp://172.20.192.66:2181>"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP_ADDR="172.20.192.66"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP_PORT="2181"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_SERVICE_HOST="172.20.192.66"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_SERVICE_PORT="2181"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_SERVICE_PORT_CLIENT="2181"
declare -x HOME="/home/zookeeper"
declare -x HOSTNAME="druid-test-zookeeper-0"
declare -x JAVA_BASE_URL="<https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jre_>"
declare -x JAVA_HOME="/usr/local/openjdk-8"
declare -x JAVA_URL_VERSION="8u232b09"
declare -x JAVA_VERSION="8u232"
declare -x JMXAUTH="false"
declare -x JMXDISABLE="false"
declare -x JMXPORT="1099"
declare -x JMXSSL="false"
declare -x KUBERNETES_PORT="<tcp://172.20.0.1:443>"
declare -x KUBERNETES_PORT_443_TCP="<tcp://172.20.0.1:443>"
declare -x KUBERNETES_PORT_443_TCP_ADDR="172.20.0.1"
declare -x KUBERNETES_PORT_443_TCP_PORT="443"
declare -x KUBERNETES_PORT_443_TCP_PROTO="tcp"
declare -x KUBERNETES_SERVICE_HOST="172.20.0.1"
declare -x KUBERNETES_SERVICE_PORT="443"
declare -x KUBERNETES_SERVICE_PORT_HTTPS="443"
declare -x LANG="C.UTF-8"
declare -x OLDPWD="/apache-zookeeper-3.5.5-bin"
declare -x PATH="/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/apache-zookeeper-3.5.5-bin/bin"
declare -x PWD="/apache-zookeeper-3.5.5-bin/bin"
declare -x SHLVL="1"
declare -x TERM="xterm"
declare -x ZK_AUTOPURGE_PURGEINTERVAL="1"
declare -x ZK_AUTOPURGE_SNAPRETAINCOUNT="3"
declare -x ZK_HEAP_SIZE="512m"
declare -x ZK_INIT_LIMIT="5"
declare -x ZK_MAX_CLIENT_CNXNS="0"
declare -x ZK_REPLICAS="3"
declare -x ZK_ROLLING_LOG_FILE_MAX_COUNT="10"
declare -x ZK_SYNC_LIMIT="2"
declare -x ZK_TICK_TIME="2000"
declare -x ZOOCFGDIR="/conf"
declare -x ZOO_ADMINSERVER_ENABLED="true"
declare -x ZOO_AUTOPURGE_PURGEINTERVAL="0"
declare -x ZOO_AUTOPURGE_SNAPRETAINCOUNT="3"
declare -x ZOO_CONF_DIR="/conf"
declare -x ZOO_DATA_DIR="/data"
declare -x ZOO_DATA_LOG_DIR="/datalog"
declare -x ZOO_INIT_LIMIT="5"
declare -x ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
declare -x ZOO_LOG_DIR="/logs"
declare -x ZOO_MAX_CLIENT_CNXNS="60"
declare -x ZOO_PORT="2181"
declare -x ZOO_STANDALONE_ENABLED="false"
declare -x ZOO_SYNC_LIMIT="2"
declare -x ZOO_TICK_TIME="2000"
looks like the vars are ?
Copy code
declare -x ZOO_AUTOPURGE_PURGEINTERVAL="0"
declare -x ZOO_AUTOPURGE_SNAPRETAINCOUNT="3"
and in the zoo.cfg file
Copy code
zookeeper@druid-test-zookeeper-0:/conf$ cat zoo.cfg
clientPort=2181
dataDir=/data
dataLogDir=/data/log
tickTime=2000
initLimit=5
syncLimit=2
maxClientCnxns=0
minSessionTimeout= 4000
maxSessionTimeout= 40000
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
4lw.commands.whitelist=*
server.1=druid-test-zookeeper-0.druid-test-zookeeper-headless.druid-cluster.svc.cluster.local:2888:3888
server.2=druid-test-zookeeper-1.druid-test-zookeeper-headless.druid-cluster.svc.cluster.local:2888:3888
server.3=druid-test-zookeeper-2.druid-test-zookeeper-headless.druid-cluster.svc.cluster.local:2888:3888
r
yes, the correct vars could be the ones from the export dump you pasted. which image are you using to deploy zk? the one given in the apache druid helm config?
j
Yeah, it's the one coming from helm chart config, no ot sure if it's 3.5.5
4 Views
Open in Slack
PreviousNext
Powered by