Hi Guys, I have 2 tables with exactly same structure created in flink-sql application .. lets say

CREATE TABLE t1_g1
(
    `id`    INT,
    `name`    STRING
)
    WITH ( 'connector' = 'kafka',
        'topic' = 't1.g1',
        'properties.auto.offset.reset' = 'earliest',
        'scan.startup.mode' = 'earliest-offset',
....       );

CREATE TABLE t1_g2
(
    `id`    INT,
    `name`    STRING
)
    WITH ( 'connector' = 'kafka',
        'topic' = 't1.g2',
        'properties.auto.offset.reset' = 'earliest',
        'scan.startup.mode' = 'earliest-offset',
....       );

Question: Is it possible or supported in flink-sql, if i can create another table that reads from both

t1.g1

and

t1.g2

topics using some wildcard or comma separated value ?? something like

CREATE TABLE t_all
(
    `id`    INT,
    `name`    STRING
)
    WITH ( 'connector' = 'kafka',
        'topic' = 't1.g1,t1.g2',
        'properties.auto.offset.reset' = 'earliest',
        'scan.startup.mode' = 'earliest-offset',
....       );
OR

OR something like with wildcard ??

CREATE TABLE t_all
(
    `id`    INT,
    `name`    STRING
)
    WITH ( 'connector' = 'kafka',
        'topic' = 't1.*',
        'properties.auto.offset.reset' = 'earliest',
        'scan.startup.mode' = 'earliest-offset',
....       );

I am pretty sure that the Kafka connector does not support that. One option would be to use a UNION ALL to read the two tables as one. (I'm not certain if there any obvious reasons not to do this. UNION would probably be tough since it is trying to avoid duplicates...)

Thanks Jim. I thought about union option.. only drawback is that it needs to know how many groups and any new addition requires an application to restart with new definition that is similar to comma separated (if supported). I believe kafka consumer supports wildcard in topic name hence just wondering if is there option available in flink-sql now or can be available in future

Makes sense. Sounds like that'd be a new feature for the Kafka connector. Maybe @Martijn Visser and/or @Tzu-Li (Gordon) Tai can weigh in.

There is https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217389320 but not yet implemented and also not scoped to Table API

Thank you guys.. i got the answer here