Hey! We noticed while trying to upgrade the iceberg connector to 1.18 that an

ARRAY<INT not null>

SQL field was previously converted to

List<Integer>

when converting to

Row

and now it seems to be an

int[]

. Does anyone have an idea what change my cause this in Flink? And whether this is expected.

Calcite is my first guess 😅

It’s a bit hard to trace the change in behaviour to a particular part in the code, the table api conversion logic is a bit thick 😄

I’ll ask around

The logical type we get seems to be both

ARRAY<INT not null>

and the only diff is in the

Row

output after converting back to stream

For the record, one of the failing Iceberg test is:

TestFlinkMetaDataTable#testAllFilesPartitioned

do you have any code snippet (would be great if flink/calcite calls only) highlighting the issue? this would probably help to answer about the cause

@Péter Váry do you have a minimal example? I personally could not repro this outside of the icberg test. But I found some other strange thing, not sure if it’s related. Consider this code:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<Tuple1<int[]>> input = env.fromElements(Tuple1.of(new int[] {1}));

        StreamTableEnvironment tableEnvironment = StreamTableEnvironment.create(env);
        Table table =
                tableEnvironment.sqlQuery(
                        "select * from " + tableEnvironment.fromDataStream(input));
        tableEnvironment.toDataStream(table).addSink(new SinkFunction<Row>() {
            @Override
            public void invoke(Row value, Context context) throws Exception {
                System.out.println(value.getField(0).getClass().getSimpleName());
            }
        });
        env.execute();

For Flink 1.17 the Row field type is

Integer[]

for Flink 1.18 it is

int[]

@Gyula Fóra: I am on PTO this week, and mostly accessing stuff from my phone, but I think the issue you have found is causing the Iceberg tests to fail too. To fix the test, I had to change this

try {
          actualArrayData = (ArrayData) actual;
        } catch (ClassCastException e) {
          actualArrayData = new GenericArrayData((Object[]) actual);
        }

To something like this:

try {
           actualArrayData = (ArrayData) actual;
         } catch (ClassCastException e) {
           actualArrayData = new GenericArrayData((int[]) actual);
         }

Notice the only change is the cast (Object->int) I am not sure where we did we see the List, and without the laptop I can not reproduce now, but based on the required test change, I expect that fixing @Gyula Fóra's example would fix the Iceberg test too.