https://flink.apache.org/ logo
Join Slack
Powered by
Hello everyone, I am trying to test flink operator...
# troubleshooting
y
Hello everyone, I am trying to test flink operator's autoscaler feature. I set this param 
kubernetes.operator.job.autoscaler.enabled: "true"
on flinkDeployment's flinkConfiguration and flink operator's flink-conf.yaml. But I think operator receive no metrics for now. And I can't find 
Autoscalar.*
related metrics every where. I have used PrometheusMetric Reporter on flink side and flink operator side for sending metrics. I am not sure how to configure Metric Collector on flink operator side. Do I need to do something to configure the metric collector pointed to Prometheus? Or the RestApiCollector will work also? 🙏
Hello everyone, I have successed to have metric of flink-operator and flink cluster both available in my prometheus, but somehow I still can't see any trace of parallelism computation from flink operator side. I have logs like these
Copy code
[flink1p-autoscaler/flink1p-autoscaler] Rest client endpoint started.
[flink1p-autoscaler/flink1p-autoscaler] Shutting down rest endpoint.
[flink1p-autoscaler/flink1p-autoscaler] Rest endpoint shutdown complete.

[INFO ][flink1p-autoscaler/flink1p-autoscaler] Resource fully reconciled, nothing to do...
flink1p-autoscaler/flink1p-autoscaler-job] Resource not found in temporary cache reading it from informer cache, for Resource ID: ResourceID{name='flink1p-autoscaler-job', namespace='flink1p-autoscaler'}
But I can't find any logs about autoscaler from scale function https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java#L82 Anyone know how to debug to understand why autoscaling is not collecting metrics and compute parallelism?
@Gyula Fóra sorry to bother you, do you have a idea how to debug this? 🙏 🙂
Does flink operator use by default API rest of flink? or we can specify to search from prometheus for example?
g
Have you tried the autoscaler example included in the repo?
y
Yes, I was using that autoscaling example, the only difference I can tell is that I have used flink 1.17.1, and it's 1.18 in example 🙂
Not sure if it is worth trying, I have upgraded flink-operator from 1.3.1 to 1.6 in K8s. @Gyula Fóra do you think it's worth to try the version 1.4 and 1.5? 🙏
g
The latest is recommended
y
but latest is 1.6? right?
latest has same tag as 1.6 here
g
Yes
🙏 1
I suggest enabling debug logs and sharing that
y
I have enabled it, I will post my logs 🙏
g
Many users have successfully used the autoscaler with operator 1.6.0 and Flink 1.17+ including us
It should work
🙏 1
y
@Gyula Fóra Here is my debug log for 1 reconciliation (flinkSessionJob and flinkDeployment ) 🙏 , please take a look when you have a moment. I guess there may just be a option forgotten somewhere but I really can't find it 🙏 What bothring me the most is • no log trace about Autoscaler (even though when I disable) • no log trace about Metric collection
operator-debug-log.txt.zip
g
could be that you have a custom operator docker file and forgot to include the autoscaler jar?
we should see some logs
y
We do have a custom operator docker file , and in the log we see
Copy code
2023-10-18 14:19:12,386 o.a.f.k.o.FlinkOperator        [INFO ]  Classpath: ./flink-kubernetes-standalone-1.6.0.jar:./flink-kubernetes-operator-1.6.0-shaded.jar:./flink-kubernetes-operator-autoscaler-1.6.0.jar:/opt/flink/operator-lib/*
So I imagine that I am with the good autoscaler jar 🙂
g
I think you are missing it actually
we are looking for somehting like
Copy code
flink-kubernetes-operator-autoscaler-1.6.0.jar
y
@Gyula Fóra in my logs (2023-10-18 141912,386 o.a.f.k.o.FlinkOperator [INFO ] Classpath: ./flink-kubernetes-standalone-1.6.0.jar./flink kubernetes operator 1.6.0 shaded.jar./flink-kubernetes-operator-autoscaler-1.6.0.jar:/opt/flink/operator-lib/*) I do have this
Copy code
:./flink-kubernetes-operator-autoscaler-1.6.0.jar
And we also have this in logs
Copy code
2023-10-18 14:19:12.989	
2023-10-18 14:19:12,989 o.a.f.k.o.u.AutoscalerLoader   [INFO ] Discovered JobAutoScaler factory: org.apache.flink.kubernetes.operator.autoscaler.JobAutoscalerFactoryImpl
means operator does find JobAutoscalerFactoryImpl , though this log only appear 1 time each day
g
I would suggest trying to install the operator with helm based on the QuickStart and trying the autoscaler example locally on minikube
That will work, and then you can start tracking down the issue
👍 1
y
Hello @Gyula Fóra, sry for asking stupid question, do you think flink-operator autoscale works for session mode deployment also, in the example I see only example with application mode
g
It will probably not work, but you can try and let us know 🙂
y
I think that might be the issue in our deployment, we use session mode
so we have CRD 
flinkDeployment
and 
flinkSessionJob
, and in example I see only 
flinkDeployment
, so looks like I need to think to test application mode ..
Hello @Gyula Fóra, just FYI, the autoscaler feature works when I switched to application mode deployment 🙏 . maybe worth to mention it in flink operator documentation so other guy like me will test it easier 😀
j
FYI, on this topic, here's what the docs say:
So it's possible to do for session jobs, it's just that there were some bugs.
y
I think it's fixed by this PR https://github.com/apache/flink-kubernetes-operator/pull/739, and we need to wait next release 1.8 of operator
🙏 1
j
Oh yea I was reading the snapshot docs for the flink operator
👍 1
5 Views
Open in Slack
PreviousNext
Powered by