Hello everyone, I'm trying to get PyFlink (v1.15) Table API to work and have encountered an issue I don't understand and would appriciate some help, if possible. I have a simple insertion query using group by, like this:

INSERT INTO sink_table
    SELECT
        `key`,
        `journeyRef`,
        MAX(`eventTimestamp`) AS `eventTimestamp`,
        SUM(`alightingCount`) AS `totalAlighting`
    FROM source_table
    GROUP BY `key`, `journeyRef`

I am trying to source from filesystem source connector to filesystem sink connector, and I am getting the error:

pyflink.util.exceptions.TableException: org.apache.flink.table.api.TableException: Table sink 'default_catalog.default_database.accjoin_filesystem' doesn't support consuming update changes which is produced by node GroupAggregate(groupBy=[key, journeyRef], select=[key, journeyRef, MAX(eventTimestamp) AS eventTimestamp, SUM(alightingCount) AS totalAlighting])

However, if I replace the sink with a print connector, it works as expected. Is this a bug, or am I missing a limitation in how the filesystem connector works?

The output of your query is a changelog stream, while the sink you're writing to is an append-only stream

I see.Thanks for you quick response. I'm wondering though, wouldn't that also be a problem for the print connector? Is it fair to assume that bounding the query in a time interval would solve the issue?

I'm wondering though, wouldn't that also be a problem for the print connector?

No, because the print connector can change the output on screen 🙂

Is it fair to assume that bounding the query in a time interval would solve the issue?

It depends 🙂 E.g. you could have late data, so that would require Flink to still take those situations into account. However, if you would use something like a Window TVF, then I believe your output is an append-only stream. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-tvf/

Interesting, I didn't know print rewrote the output. TIL. Thanks for the pointers, I will take a closer look at wTVF. It looks promissing.