I'm facing an issue while writing to parquet, my schema for table t(c1 array<row<x string>>), input data is in json:

{"c1" : [{"x":"1"}, {"x":"2"}]}

, I'm getting issue, caused by:

Caused by: org.apache.parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead
	at org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:329)
	at org.apache.flink.formats.parquet.row.ParquetRowDataWriter$ArrayWriter.write(ParquetRowDataWriter.java:420)
	at org.apache.flink.formats.parquet.row.ParquetRowDataWriter$RowWriter.write(ParquetRowDataWriter.java:456)
	at org.apache.flink.formats.parquet.row.ParquetRowDataWriter.write(ParquetRowDataWriter.java:79)
	at org.apache.flink.formats.parquet.row.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:89)

but if input data is {"c1": []} , empty array, then parquet write is successful, but if my schema is simple array without row that is table t(c1 array<p string>), it is working fine, is there some issue with array<row> data type

I found the issue, but unable to understand why method

public void write(ArrayData arrayData, int ordinal) {}

is not implemented in RowWriter, here in ParquetRowDataWriter class, https://github.com/apache/flink/blob/master/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java#L477

its better to send this in the mailing list

which mailing list