https://flink.apache.org/ logo
Join Slack
Powered by
Hi I am using using Flink K8s operator 1.4 with Fl...
# troubleshooting
m
Hi I am using using Flink K8s operator 1.4 with Flink 1.17. I am using upgradeMode: last-state with checkpointing enabled which are written to GCS bucket. What I am observing is when the job autoscales the job is not restoring from the last checkpoint before the autoscaling. Even the config-map to which the checkpoints get stored is recreated after the job autoscales. Has anyone faced a similar issue ?
g
Please open a jira ticket with the relevant operator logs and the yaml of the resource
m
@Gyula Fóra Sure I will raise the JIRA. I tried with this autoscaling example https://github.com/apache/flink-kubernetes-operator/tree/release-1.4/examples/autoscaling also. Observed a similar behaviour. By any chance, have you tried an autoscaling job that restores from checkpoints after autoscaling ? It would be clear if it is Operator bug or some kind of setup issue of mine.
@Gyula Fóra It was a permission issue on a directory at my end. But now what I am observing is my job moves to SUCCESSFUL state and never autoscales. I am seeing this in job manager logs: 
2023-03-06 13:09:20,680 INFO  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap [] - Application completed SUCCESSFULLY
Operator logs: 
2023-03-06 13:08:24,528 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Event  | Info    | SCALINGREPORT   | Scaling vertices: Vertex ID bc764cd8ddf7a0cff126f51c16239658 | Parallelism 25 -> 167 | Processing capacity 286382.74 -> 22472349.00 | Target data rate 60383.55 Vertex ID 0a448493b4782967b150582570326227 | Parallelism 20 -> 400 | Processing capacity 220176.24 -> 44944708.00 | Target data rate 120767.12
2023-03-06 13:08:24,743 o.a.f.k.o.c.FlinkDeploymentController [INFO ][default/replication-2] End of reconciliation
2023-03-06 13:08:24,745 o.a.f.k.o.c.FlinkDeploymentController [INFO ][default/replication-2] Starting reconciliation
2023-03-06 13:08:24,746 o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][default/replication-2] Getting service for replication-2
2023-03-06 13:08:24,746 o.a.f.k.o.o.JobStatusObserver  [INFO ][default/replication-2] Observing job status
2023-03-06 13:08:24,761 o.a.f.k.o.o.JobStatusObserver  [INFO ][default/replication-2] Job status (RUNNING) unchanged
2023-03-06 13:08:24,767 o.a.f.k.o.c.FlinkConfigManager [INFO ][default/replication-2] Generating new config
2023-03-06 13:08:24,773 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][default/replication-2] UPGRADE change(s) detected (FlinkDeploymentSpec[job.upgradeMode=SAVEPOINT,flinkConfiguration.pipeline.jobvertex-parallelism-overrides=0a448493b4782967b150582570326227:400,bc764cd8ddf7a0cff126f51c16239658:167] differs from FlinkDeploymentSpec[job.upgradeMode=STATELESS,flinkConfiguration.pipeline.jobvertex-parallelism-overrides=<null>]), starting reconciliation.
2023-03-06 13:08:24,837 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Event  | Info    | SPECCHANGED     | UPGRADE change(s) detected (FlinkDeploymentSpec[job.upgradeMode=SAVEPOINT,flinkConfiguration.pipeline.jobvertex-parallelism-overrides=0a448493b4782967b150582570326227:400,bc764cd8ddf7a0cff126f51c16239658:167] differs from FlinkDeploymentSpec[job.upgradeMode=STATELESS,flinkConfiguration.pipeline.jobvertex-parallelism-overrides=<null>]), starting reconciliation.
2023-03-06 13:08:24,837 o.a.f.k.o.r.d.AbstractJobReconciler [INFO ][default/replication-2] Upgrading/Restarting running job, suspending first...
2023-03-06 13:08:24,837 o.a.f.k.o.r.d.AbstractJobReconciler [INFO ][default/replication-2] Job is in running state, ready for upgrade with SAVEPOINT
2023-03-06 13:08:24,848 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Event  | Info    | SUSPENDED       | Suspending existing deployment.
2023-03-06 13:08:24,849 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Suspending job with savepoint.
2023-03-06 13:08:26,388 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Job successfully suspended with savepoint <gs://flame-poc/savepoints/savepoint-a5efca-ac0a07066334>.
2023-03-06 13:08:26,390 o.a.f.k.o.c.FlinkDeploymentController [INFO ][default/replication-2] End of reconciliation
2023-03-06 13:08:26,436 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Status | Info    | UPGRADING       | The resource is being upgraded
2023-03-06 13:08:26,438 o.a.f.k.o.c.FlinkDeploymentController [INFO ][default/replication-2] Starting reconciliation
2023-03-06 13:08:26,439 o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][default/replication-2] Getting service for replication-2
2023-03-06 13:08:26,442 o.a.f.k.o.o.d.ApplicationObserver [WARN ][default/replication-2] Running deployment generation 2 doesn't match upgrade target generation 3.
2023-03-06 13:08:26,443 o.a.f.k.o.c.FlinkConfigManager [INFO ][default/replication-2] Generating new config
2023-03-06 13:08:26,444 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][default/replication-2] UPGRADE change(s) detected (FlinkDeploymentSpec[job.state=RUNNING] differs from FlinkDeploymentSpec[job.state=SUSPENDED]), starting reconciliation.
2023-03-06 13:08:26,486 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Status | Info    | UPGRADING       | The resource is being upgraded
2023-03-06 13:08:26,487 o.a.f.k.o.r.d.ApplicationReconciler [INFO ][default/replication-2] Deleting deployment with terminated application before new deployment
2023-03-06 13:08:26,487 o.a.f.k.o.s.NativeFlinkService [INFO ][default/replication-2] Deleting JobManager deployment and HA metadata.
2023-03-06 13:08:30,576 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Waiting for cluster shutdown... (5s)
2023-03-06 13:08:35,631 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Waiting for cluster shutdown... (10s)
2023-03-06 13:08:38,659 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Cluster shutdown completed.
2023-03-06 13:08:38,659 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Deleting Kubernetes HA metadata
2023-03-06 13:08:38,675 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Cluster shutdown completed.
2023-03-06 13:08:38,697 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Event  | Info    | SUBMIT          | Starting deployment
2023-03-06 13:08:38,697 o.a.f.k.o.s.AbstractFlinkService [INFO ][default/replication-2] Deploying application cluster
2023-03-06 13:08:38,702 o.a.f.k.o.s.NativeFlinkService [INFO ][default/replication-2] Deploying application cluster
2023-03-06 13:08:38,702 o.a.f.c.d.a.c.ApplicationClusterDeployer [INFO ][default/replication-2] Submitting application in 'Application Mode'.
2023-03-06 13:08:38,760 o.a.f.k.u.KubernetesUtils      [INFO ][default/replication-2] Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124
2023-03-06 13:08:38,760 o.a.f.k.u.KubernetesUtils      [INFO ][default/replication-2] Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122
2023-03-06 13:08:38,760 o.a.f.k.u.KubernetesUtils      [INFO ][default/replication-2] Kubernetes deployment requires a fixed port. Configuration high-availability.jobmanager.port will be set to 6123
2023-03-06 13:08:38,760 o.a.f.c.Configuration          [INFO ][default/replication-2] Config uses fallback configuration key 'kubernetes.pod-template-file' instead of key 'kubernetes.pod-template-file.jobmanager'
2023-03-06 13:08:38,762 o.a.f.c.Configuration          [INFO ][default/replication-2] Config uses fallback configuration key 'kubernetes.service-account' instead of key 'kubernetes.jobmanager.service-account'
2023-03-06 13:08:38,765 o.a.f.c.Configuration          [INFO ][default/replication-2] Config uses fallback configuration key 'kubernetes.pod-template-file' instead of key 'kubernetes.pod-template-file.taskmanager'
2023-03-06 13:08:38,765 o.a.f.c.Configuration          [INFO ][default/replication-2] Config uses fallback configuration key 'kubernetes.pod-template-file' instead of key 'kubernetes.pod-template-file.taskmanager'
2023-03-06 13:08:38,830 o.a.f.k.KubernetesClusterDescriptor [WARN ][default/replication-2] Please note that Flink client operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been set to ClusterIP.
2023-03-06 13:08:38,831 o.a.f.k.KubernetesClusterDescriptor [INFO ][default/replication-2] Create flink application cluster replication-2 successfully, JobManager Web Interface: <http://replication-2-rest.default:8081>
2023-03-06 13:08:38,833 o.a.f.k.o.s.NativeFlinkService [INFO ][default/replication-2] Application cluster successfully deployed
2023-03-06 13:08:38,834 o.a.f.k.o.c.FlinkDeploymentController [INFO ][default/replication-2] End of reconciliation
2023-03-06 13:08:38,896 o.a.f.k.o.l.AuditUtils         [INFO ][default/replication-2] >>> Status | Info    | DEPLOYED        | The resource is deployed/submitted to Kubernetes, but it's not yet considered to be stable and might be rolled back in the future
2023-03-06 13:08:38,897 o.a.f.k.o.c.FlinkDeploymentController [INFO ][default/replication-2] Starting reconciliation
2023-03-06 13:08:38,898 o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][default/replication-2] Getting service for replication-2
2023-03-06 13:08:38,898 o.a.f.k.o.o.d.ApplicationObserver [INFO ][default/replication-2] Observing JobManager deployment. Previous status: DEPLOYING
2023-03-06 13:08:38,912 o.a.f.k.o.o.d.ApplicationObserver [INFO ][default/replication-2] JobManager is being deployed
2023-03-06 13:08:38,913 o.a.f.k.o.a.JobAutoScalerImpl  [INFO ][default/replication-2] Job autoscaler is waiting for RUNNING job state
20
2 Views
Open in Slack
PreviousNext
Powered by