Hi team, I am facing a weird issue while using `Fl...
# troubleshooting
r
Hi team, I am facing a weird issue while using
Flink v1.19.2
. I am trying to read an s3 path using the
FileSystem connector
. If I try it the
Table API Connector
way, the read happens the first time but subsequently if new "partitions" are getting added, Flink is not aware of them even when I provide
'source.monitor-interval'
to make it unbounded.
Copy code
String createTableSql = """
    CREATE TABLE tbl (
        col1 STRING,
        part_col STRING
    ) PARTITIONED BY (part_col)
    WITH (
        'connector' = 'filesystem',
        'path' = 's3a:/bucket_name/path',
        'format' = 'parquet',
        'source.monitor-interval' = '10s'
    )
""";
However if I use the
Datastream
Connector, which is low level and not something that I want to use, it works if new partition folders are added. I don't see any error logs even when I enabled DEBUG logs. Can someone help with this? I am suspecting that the partition discovery isn't working.
p
how does your partition directories looks like? The naming I mean
r
Copy code
path
└── part_col=1745567815918
    ├── part-0.snappy.parquet
    ├── _SUCCESS_
└── part_col=1745567815920
    ├── part-0.snappy.parquet
    ├── _SUCCESS_
p
I was going to point out about the directory names and the presence of
=
. One last question for me. When you create your environment do you set
inStreamingMode
?
r
Hi @Pedro Mázala , yes here is my code. Also I don't understand "regarding the directory names and the presence of `=`"
Copy code
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

EnvironmentSettings settings = EnvironmentSettings.inStreamingMode();

StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);

String createTableSql = """
    CREATE TABLE tbl (
        col1 STRING,
        part_col STRING
    ) PARTITIONED BY (part_col)
    WITH (
        'connector' = 'filesystem',
        'path' = 's3a:/bucket_name/path',
        'format' = 'parquet',
        'source.monitor-interval' = '10s'
    )
""";
tableEnv.executeSql(createTableSql);
String selectQuery = """
    SELECT * FROM tbl
    /*+ OPTIONS('source.monitor-interval'='10s') */
""";

Table table = tableEnv.sqlQuery(selectQuery);
DataStream<Row> dataStream = tableEnv.toDataStream(table);
DataStream<String> output = dataStream.map(new MapFunction<Row, String>() {
    @Override
    public String map(Row row) {
        return "Processed: " + row.toString();
    }
});

output.addSink(new DiscardingSink<>()).name("Discard");
env.execute("Flink Parquet Stream Reader");
Hi @Pedro Mázala anything on this 🤔
p
Hey there! I did not managed to check it yet.
🙏 1