Has been a pretty good experience I’ll say, especially with pointing Zeppelin at Flink session clusters orchestrated by Flink Operator.

And how do you presist all the tables or data in case of restart ?

( Any tips or suggsetions)

Found it https://zeppelin.apache.org/docs/latest/interpreter/flink.html. Any suggestions you can let me know 🙂

how did you point zeppline to flink session Cluster ?

via

flink.execution.remote.host

and

flink.execution.remote.port

Table wise we’re on Amazon so we use AWS Glue Catalog as a sort of hive metastore @Rajat Ahuja

so do we need SQL gateway to use zeppline ? and @sap1ens do you run zeppline on a different pod or in the same container as job manager ?

I run it in a different deployment

is that correct ? and any suggestions for metastore if it is to be done on on-prem ( not cloud) ?

Hive is still a typical option here, however we don't use a metastore for Flink/Zeppelin

Thanks for pointers, i could set up zeppline on k8s, and create roles and role bindings and then changed the properties to run the job. Here are the files but somehow when interpreters pod comes up following error. Not sure how can i provide slf4j and other jars to intrepreter

Interpreter download command: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///opt/zeppelin/conf/log4j.properties -Dlog4j.configurationFile=file:///opt/zeppelin/conf/log4j2.properties -Dzeppelin.log.file=/opt/zeppelin/logs/zeppelin-interpreter-flink-shared_process--flink-xajzxr.log -cp :/lib/*.jar:::/opt/zeppelin/interpreter/zeppelin-interpreter-shaded-0.10.0.jar:: org.apache.zeppelin.interpreter.remote.RemoteInterpreterDownloader zeppelin-server.apl-edm-streaming-svcs-l3-dev1.svc 32854 flink /tmp/local-repo/flink
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterDownloader.<clinit>(RemoteInterpreterDownloader.java:35)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 1 more

And on zeppline i see

My set up ::

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zeppelin-flink-testing
  namespace: apl-edm-streaming-svcs-l3-dev1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zeppelin-flink-testing
  template:
    metadata:
      labels:
        app: zeppelin-flink-testing
    spec:
      serviceAccountName: zep-flink-svc
      containers:
        - name: zeppelin
          image: artifactory.us.bank-dns.com:5000/flink/zeppelin:0.10.0
          ports:
            - containerPort: 8080
      imagePullSecrets:
        - name: usb-artifactory-prod1
      volumes:
        - name: zeppelin-notebook
          emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: zeppelin-flink-testing-service
  namespace: apl-edm-streaming-svcs-l3-dev1
spec:
  selector:
    app: zeppelin-flink-testing
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
          
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: apl-edm-streaming-svcs-l3-dev1
  name: zep-flink-role
rules:
  - apiGroups: [""]
    resources: ["services", "pods"]   # Add "pods" resource here
    verbs: ["get", "list", "create", "watch", "delete"]  # Add "create" verb here
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: zep-flink-svc
  namespace: apl-edm-streaming-svcs-l3-dev1 # Replace with the desired namespace where you want to create the service account

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zep-flink-rolebinding
  namespace: apl-edm-streaming-svcs-l3-dev1
subjects:
  - kind: ServiceAccount
    name: zep-flink-svc
    namespace: apl-edm-streaming-svcs-l3-dev1
roleRef:
  kind: Role
  name: zep-flink-role
  apiGroup: rbac.authorization.k8s.io

@M C (Mac) @sap1ens any suggestions how can i fix the error and any pointers what can i better do to productnize ?

Seems like need to include flink lib jars + logging jars manually via flink.exection.packages

10.0.X only supports up to Flink 1.14.x to which if they’re included in the container you’ll be able to chat remotely and set up a local cluster for the notebook as well

For your particular error, for 10.0.X the remote interpreter stuff was broken I was suppose to file a bug with some detail. Remote interpreter stuff being Zeppelin on kubernetes can run a process in the same container as the notebook, or dynamically spawn pods and delegate cells to execute there. The pod spin up was fine for me, but the actual remote execution logic/bash scripts/jars was broken and couldn’t do simple Python like declaring a variable for example 😅

If you want to tag team on reporting that issue let me know

Context we’re on EKS 1.24 or 25 I forget, but containerd is the runtime, I’ve also seen this on minikube

@sap1ens can i connect zeppelin to SQL gateway and that way we can avoid connectors jars required and push all connectors jars to SQL gateway ?

Hi folks, i built the docker image "FROM openjdk:8 as builder ADD . /workspace/zeppelin WORKDIR /workspace/zeppelin ENV MAVEN_OPTS="-Xms1024M -Xmx2048M -XX:MaxMetaspaceSize=1024m -XX:-UseGCOverheadLimit -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn" # Allow npm and bower to run with root privileges RUN echo "unsafe-perm=true" > ~/.npmrc && \ echo '{ "allow_root": true }' > ~/.bowerrc && \ ./mvnw -B package -DskipTests -Pbuild-distr -Pspark-3.2 -Pinclude-hadoop -Phadoop3 -Pspark-scala-2.12 -Pweb-angular -Pweb-dist && \ # Example with doesn't compile all interpreters # ./mvnw -B package -DskipTests -Pbuild-distr -Pspark-3.2 -Pinclude-hadoop -Phadoop3 -Pspark-scala-2.12 -Pweb-angular -Pweb-dist -pl '!groovy,!submarine,!livy,!hbase,!file,!flink' && \ mv /workspace/zeppelin/zeppelin-distribution/target/zeppelin-/zeppelin- /opt/zeppelin/ && \ # Removing stuff saves time, because docker creates a temporary layer rm -rf ~/.m2 && \ rm -rf /workspace/zeppelin/* FROM ubuntu:20.04 COPY --from=builder /opt/zeppelin /opt/zeppelin" from 1.17.1 branch but i guess it just builds the base image of zeppelin. does any one know the next steps ? Also for zeppelin to run in remote mode do we still need a flink library ?

If you need some custom dependencies, you need to put them into

${ZEPPELIN_HOME}/local-repo/flink/

, I found that other ways don't work.