Raj Kunwar Singh
04/29/2025, 3:04 PMFlink v1.19.2. I am trying to read an s3 path using the FileSystem connector.
If I try it the Table API Connector way, the read happens the first time but subsequently if new "partitions" are getting added, Flink is not aware of them even when I provide 'source.monitor-interval' to make it unbounded.
String createTableSql = """
CREATE TABLE tbl (
col1 STRING,
part_col STRING
) PARTITIONED BY (part_col)
WITH (
'connector' = 'filesystem',
'path' = 's3a:/bucket_name/path',
'format' = 'parquet',
'source.monitor-interval' = '10s'
)
""";
However if I use the Datastream Connector, which is low level and not something that I want to use, it works if new partition folders are added.
I don't see any error logs even when I enabled DEBUG logs. Can someone help with this? I am suspecting that the partition discovery isn't working.Pedro Mázala
04/29/2025, 5:20 PMRaj Kunwar Singh
04/30/2025, 12:42 AMpath
└── part_col=1745567815918
├── part-0.snappy.parquet
├── _SUCCESS_
└── part_col=1745567815920
├── part-0.snappy.parquet
├── _SUCCESS_Pedro Mázala
04/30/2025, 12:00 PM=.
One last question for me. When you create your environment do you set inStreamingMode?Raj Kunwar Singh
04/30/2025, 1:09 PMfinal StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
EnvironmentSettings settings = EnvironmentSettings.inStreamingMode();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);
String createTableSql = """
CREATE TABLE tbl (
col1 STRING,
part_col STRING
) PARTITIONED BY (part_col)
WITH (
'connector' = 'filesystem',
'path' = 's3a:/bucket_name/path',
'format' = 'parquet',
'source.monitor-interval' = '10s'
)
""";
tableEnv.executeSql(createTableSql);
String selectQuery = """
SELECT * FROM tbl
/*+ OPTIONS('source.monitor-interval'='10s') */
""";
Table table = tableEnv.sqlQuery(selectQuery);
DataStream<Row> dataStream = tableEnv.toDataStream(table);
DataStream<String> output = dataStream.map(new MapFunction<Row, String>() {
@Override
public String map(Row row) {
return "Processed: " + row.toString();
}
});
output.addSink(new DiscardingSink<>()).name("Discard");
env.execute("Flink Parquet Stream Reader");Raj Kunwar Singh
05/01/2025, 1:33 PMPedro Mázala
05/01/2025, 2:03 PM