Hi, I am trying to collect files from HDFS in my ...
# troubleshooting
c
Hi, I am trying to collect files from HDFS in my DataStream job. I need to collect two types of files - CSV and Parquet. I understand that Flink supports both formats but in Streaming mode, Flink doesnt support splitting these formats. Splitting is only supported in Table API. I wanted to understand the thought process around this and why splitting is not supported in CSV and AvroParquet Stream formats? As far as my understanding goes, splitting would work fine with HDFS blocks and multiple blocks can be read in parallel. Maybe I am missing some fundamental aspect about this. Would like to understand more if someone can point me in the right direction. Thanks