StreamSource, then for each entity from streaming ...
# random
s
StreamSource, then for each entity from streaming I need to do table queries (using query parameters from entity), then union/minus/intersect tables, and resulting table write to StreamSink. Do you know how to create TableEnvironment in Batch mode from StreamExectutionEnvironment? Because StreamTableEnvironment does not support union operation
d
What version of Flink are you using, and do you actually need the StreamExecutionEnvironment (i.e., are you using both the DataStream and Table APIs)?
The BatchTableEnvironment was removed in 1.14. You can either use the batch/streaming unified TableEnvironment, or a StreamExecutionEnvironment running in batch execution mode.
s
I'm using Flink 1.15.1. Could you tell me please, how can I create TableEnvironment.inBatchMode() out of StreamExecutionEnvironment? When I do TableEnvironment.create(env), it creates StreamTableEnvironment. I do TableBatch processing inside of flatmap transformation of streaming, and I need to work with tables in batch mode (do intersection, union and minus operations of Tables)
d
I would try this:
Copy code
env.setRuntimeMode(RuntimeExecutionMode.BATCH);
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
I think that should work. Let me know what you find.
s
@David Anderson I mean I need StreamExecutionEnvironment running in Stream mode, but only TableEnvironment running in batch mode
d
I don’t believe that’s possible. Maybe run the batch table job first and then consume its results in the streaming job? You might find the Table Store a convenient place to store the intermediate results.
s
@David Anderson I can't run the BatchTable job first, because first I need to get data from StreamSource, then for each entity from streaming I need to do table queries (using query parameters from entity), then union/minus/intersect tables, and resulting table write to StreamSink
Unfortunately StreamTableEnvironment does not support union of 2 tables