Hi, While Python Table API and creating table using watermark the watermark strategy is not working, but while using sql-client the same table query is working fine. Can any please help me out to know what might be the potential issue

What's the difference between how you defined the watermark in Python vs in SQL?

None in Python file i am using table streaming enviorment . env_settings = EnvironmentSettings.in_streaming_mode() t_env = TableEnvironment.create(env_settings) and using t_env.execute_sql(“Create table watermark watermark startegy”) Here when using same sql in sql-client it is working fine.

t_env.execute_sql(“Create table watermark watermark startegy”)

You put your SQL statement there, I suppose?

yes

If you have something like

source_table_ddl = f"""
            CREATE TABLE {source_table} (
                a TINYINT,
                b SMALLINT,
                rowtime TIMESTAMP(3),
                WATERMARK FOR rowtime AS rowtime - INTERVAL '60' MINUTE
            ) WITH (
                'connector' = 'filesystem',
                'path' = '{source_path}',
                'format' = 'csv'
            )
        """
        self.t_env.execute_sql(source_table_ddl)

It should work

yes this is what my create statement looks like CREATE OR REPLACE TABLE Test ( event_time bigint, event_name string, event_time_ltz as TO_TIMESTAMP_LTZ(eventTime,3),

rowtime

TIMESTAMP(3) METADATA FROM ‘timestamp’, WATERMARK FOR event_time_ltz AS event_time_ltz - INTERVAL ‘5’ SECOND )WITH ( ‘connector’ = ‘kafka’, ‘topic’ = ‘{topic_name}’, ‘properties.bootstrap.servers’ = ‘{kafka_brokers}’, ‘properties.group.id’ = ‘test-consumer-group’, ‘format’ = ‘json’, ‘scan.startup.mode’ = ‘latest-offset’, ‘json.fail-on-missing-field’ = ‘false’ );

There's no

CREATE OR REPLACE TABLE

syntax in Flink

okay let me try

https://issues.apache.org/jira/browse/FLINK-32515 is the open ticket for adding support on this

No Watermark (Watermarks are only available if EventTime is used) Still showing in UI

Yeah but that only changes if you're actually reading data

How much data is there inside the topic and how many partitions does it have?

Same source but using sql, the watermark is working as expected, 0 1688482240285 04/07/2023, 202040

One is running with parallelism of 20, the other with 8. There is a difference in your setup

Yeah but parallelism should not matter the watermark

Parallelism determines potential idleness of your topic

Try setting the same parallelism for both jobs

@Martijn Visser - do we have any documentation to understand more about it

Cool thanks yes with parallelism 8 it is working great. THanks

https://medium.com/@ipolyzos_/understanding-watermarks-in-apache-flink-c8793a50fbb8 This might help a little bit.

Thank @Martijn Visser for helping me out.