Flink SQL API: Let's assume two table that im running a join using sql.. when using rocksdb state backend i see that there are 2 column families being created - whether its an inner join left join etc -

right-records

and

left-records

. 1. Is there any way to specify the name for the column families? How are these names assigned? 2. Do we always keep both tables in state? for example if the first table is an append only stream how does state get expired in sql? if i have billions of events how do i make sure they dont stay around and fill up the disk?

Are you referring to a regular join with 2?

Yes I tested with an Inner Join

Then yes, it will be kept in state forever. That's also documented at https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#regular-joins

so from a quick look when i use sql having a realtime stream and an append only stream (for state) i need to use ``table.exec.state.ttl` to make sure all the records for the the realtime stream are not kept in memory or in rocksdb, correct?

That ultimately depends on your requirements of course, because including a TTL could impact the correctness of your query. But yes, you set a TTL to avoid that records will be kept in state forever

Great, thanks 😄

With regards to naming, you should be able to use

SELECT p.price AS RealPrice FROM price p

the naming is for the column families in rocksdb.. i.e

SELECT * 
FROM readings
  LEFT JOIN sensors ON readings.id = sensors.id

if i run this join with rocksdb as the statebackend it will create two column families for each stream with names

right-records

and

left-records

. And i was curious if there was any way to specify the names of the column families

I doubt you can control that. In the end, the SQL is optimized and transformed into an application by the planner which doesn't offer fined grained control over these type of things