Hello Everyone, We’ve developed an application wit...
# troubleshooting
m
Hello Everyone, We’ve developed an application with the following workflow: • Streaming data from Kafka • Performing UPSERT operations (insert/update/delete) into a Table API object after filtering • Executing a SQL join across approximately 12 Table API views • Emitting an event with the SQL results only if the join conditions are satisfied Questions: 1. How can I debug the contents of the internal Flink tables when the SQL join conditions are not met? 2. Does Flink provide any built-in API to side output records in such scenarios? 3. Is there an API available to dynamically query all 12 tables on the fly for a specific key? Could you please let me know your view on this questions @David Anderson
d
Using the built-in join operator, there's not much support for any of the points you've raised. The internal state cannot be inspected, nor can you do an ad-hoc query against those tables. As for side outputs, what you can sometimes do is to create a statement set that includes both the primary query and a secondary query with the opposite condition. The Flink SQL planner will co-optimize the two queries, having them share as much as possible of the early stages of both pipelines, and then fork the data streams as needed to produce different outputs. Depending on the details, this may or may not be a useful technique. A possible alternative would be to use a PTF instead of the built-in join(s). A single ProcessTableFunction can have 12 input tables, and since you'd be managing all the state yourself, you could make it inspectable or queryable.
m
Thank you @David Anderson will try PTF and update the status. On a different note, we are using RocksDBStateBackend for stateStore, where stateStore can grow rapidly, please let us know the efficient way to query and purge the contents of statestore like delete last two month old records or records that have payload status as 'completed'
d
An "obvious" answer is to use state TTL. Note that state TTL is always in terms of processing time, and not event time. There may also be ways you can adjust the query so that it needs less state. The details will depend on the nature of the streams involved (whether they are append-only, upsert, or retract streams), and which join operations are being used.

https://www.youtube.com/watch?v=LOmHQvYFQdw▾

covers a lot of what you need to know about this subject.
👍 1
m
Appreciate your help, thank you