Hi All! I have a question regarding savepoints in ...
# troubleshootingr
Richard Noble
02/02/2023, 1:58 PMHi All! I have a question regarding savepoints in flink SQL.
Is there a way to identify a SQL clause as an operator? So that if I recover from a savepoint, that query runs from that point on?
m
Martijn Visser
02/02/2023, 2:01 PMI'm not sure I understand your use case, but you can already start a SQL job from your savepoint. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#start-a-sql-job-from-a-savepoint
r
Richard Noble
02/02/2023, 2:06 PMI get that bit, when I did though, I got:
Copy code
java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint <s3://flink-checkpoints-prod-us/savepoints/savepoint-f24148-365a16502fb2>. Cannot map checkpoint/savepoint state for operator 81b0d5817065b956d863390ee24c34a0 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.Richard Noble
02/02/2023, 2:08 PMWhich I believe was due t modifying the query. However, in my case I know that it'd be safe to recover specific SQL statements, within the statement set.
Now, as I understand it, in SQL, the operator ID's are generated.
My question was, if there was a way to 'point' a query to an operator, so that it'd recover from that point?
m
Martijn Visser
02/02/2023, 2:10 PMYes, query modifications can result in an incompatible plan. Upgrades from those are currently not supported
r
Richard Noble
02/02/2023, 2:18 PMThanks, is there a recommended pattern / workaround for upgrading, or just destroy and replay?
m
Martijn Visser
02/02/2023, 2:24 PMCurrently it's destroy and replay. I could envision that at some point patterns from FLIP-190 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 could be extended to support these breaking changes, but that's just an idea in my head
r
Richard Noble
02/02/2023, 2:47 PMThanks again, much apprecaited!
a
Ans Fida
02/02/2023, 6:41 PMIn a scenario like this, how would the
allowNonRestoredStatem
Martijn Visser
02/02/2023, 6:43 PMIt won’t work because you can’t specify it in a Flink SQL client
a
Ans Fida
02/02/2023, 7:30 PMI see, how would it work for the normal flink client (cli client)? E.g., If we run a job (submit a jar) with an updated SQL query and add the
allowNonRestoredStatem
Martijn Visser
02/03/2023, 8:21 AMYou mean from a Table API application JAR where you use SQL?
a
Ans Fida
02/03/2023, 5:35 PMyep
m
Martijn Visser
02/03/2023, 5:37 PMProbably wouldn’t work. That option is designed for removing an operator from a plan, not for changing an operator
Martijn Visser
02/03/2023, 5:38 PMBut not 100% sure
2 Views