Hi everyone. I"m facing an issue with convert data...
# troubleshooting
l
Hi everyone. I"m facing an issue with convert data stream to table
Copy code
DataStream<Row> merchantQuasiFeatureCreditFilterRow = merchantQuasiFeatureCreditFilter
        .map(message -> Row.of(
            message.getEntity(),        // Field 0
            message.getFeatureSet(),    // Field 1
            message.getTimestamp()    ,  // Field 2
            sha256(message.getEntity())  // Field 3
        ))
        .returns(Row.class);
    tableEnv.createTemporaryView("datastream_table", merchantQuasiFeatureCreditFilterRow,
    Schema.newBuilder()
            .columnByExpression("entity", "cast(f0 as string)")
            .columnByExpression("proctime", "PROCTIME()") //
            .columnByExpression("entityHash", "cast(f3 as string)") // Hashed entity
        .build()
    );
It show error when deploy Flink job:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Invalid expression for computed column 'entityHash'.
I don't know why because I've already add field 3 to data stream.
p
Have you tried something like:
Copy code
Schema schema = Schema.newBuilder()
        .column("entity",      DataTypes.STRING())          
        .column("featureSet",  DataTypes.STRING())          
        .column("ts",          DataTypes.BIGINT())          
        .column("entityHash",  DataTypes.STRING())          
        .columnByExpression("proctime", "PROCTIME()")
        .build();

tableEnv.createTemporaryView(
        "datastream_table",
        merchantQuasiFeatureCreditFilterRow,
        schema);
🙏 1
l
it show another error: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to find a field named 'entity' in the physical data type derived from the given type information for schema declaration. Make sure that the type information is not a generic raw type. Currently available fields are: [f0]
Copy code
Schema schema = Schema.newBuilder()
            .column("entity",      DataTypes.STRING())
            .column("ts",          DataTypes.BIGINT())
            .column("entityHash",  DataTypes.STRING())
            .columnByExpression("proctime", "PROCTIME()")
            .build();

    tableEnv.createTemporaryView(
            "datastream_table",
            merchantQuasiFeatureCreditFilterRow,
            schema);

    tableEnv.from("datastream_table").printSchema();

    Table result = tableEnv.sqlQuery(
        "SELECT d.*, h.features.merchant_address_city " +
            "FROM datastream_table AS d " +
            "LEFT JOIN merchant_feature_hbase FOR SYSTEM_TIME AS OF d.proctime AS h " +
            "ON d.entity = h.entity" // Use hashed entity for join
    );
should I change d.entity to d.f0?
p
In which line does this happen? I think probably on
tableEnv.createTemporaryView
line, right?
l
yes I think it happen in createTemporaryView
I tried many different syntax and GPT but still error.
p
Did you try mapping your datastream output to a named row instead of a row?
l
do you mean create a custom POJO class and return something like this DataStream<StreamingData> merchantQuasiFeatureCreditFilterData = merchantQuasiFeatureCreditFilter .map(message -> new StreamingData( message.getEntity(), message.getTimestamp(), sha256(message.getEntity()) ));
p
not really. something like
Copy code
TypeInformation<Row> merchantType = Types.ROW_NAMED(
        new String[] {"entity", "featureSet", "ts", "entityHash"},
        Types.STRING, Types.STRING, Types.LONG, Types.STRING);   // adjust ts type if needed

DataStream<Row> merchantQuasiFeatureCreditFilterRow =
        merchantQuasiFeatureCreditFilter
            .map(msg -> Row.of(
                    msg.getEntity(),
                    msg.getFeatureSet(),
                    msg.getTimestamp(),
                    sha256(msg.getEntity())))
            .returns(merchantType);
🙏 1
l
i will try your suggestion. I'm trying something similar: TypeInformation<Row> rowTypeInfo = new RowTypeInfo( Types.STRING, // f0 - entity Types.LONG, // f1 - timestamp Types.STRING // f2 - hashed entity ); DataStream<Row> merchantQuasiFeatureCreditFilterRow = merchantQuasiFeatureCreditFilter .map(message -> Row.of( message.getEntity(), // Field 0 message.getTimestamp(), // Field 1 sha256(message.getEntity()) // Field 2 )) .returns(rowTypeInfo); // Use explicit type information instead of Row.class
Just try this and it work. I think it similar with create a pojo class
Copy code
TypeInformation<Row> rowTypeInfo = new RowTypeInfo(
                Types.STRING,  // f0 - entity
                Types.LONG,    // f1 - timestamp
                Types.STRING   // f2 - hashed entity
        );


        DataStream<Row> merchantQuasiFeatureCreditFilterRow = merchantQuasiFeatureCreditFilter
                .map(message -> Row.of(
                        message.getEntity(),        // Field 0
                        message.getTimestamp(),     // Field 1
                        sha256(message.getEntity()) // Field 2
                ))
                .returns(rowTypeInfo); // Use explicit type information instead of Row.class

        Schema schema = Schema.newBuilder()
                .column("f0", DataTypes.STRING())
                .column("f1", DataTypes.BIGINT())
                .column("f2", DataTypes.STRING())
                .columnByExpression("proctime", "PROCTIME()")
                .build();

        tableEnv.createTemporaryView(
                "datastream_table",
                merchantQuasiFeatureCreditFilterRow,
                schema);

        tableEnv.from("datastream_table").printSchema();
        Table result = tableEnv.sqlQuery(
                "SELECT d.*, h.features.merchant_address_city " +
                        "FROM datastream_table AS d " +
                        "LEFT JOIN merchant_feature_hbase FOR SYSTEM_TIME AS OF d.proctime AS h " +
                        "ON d.f2 = h.entity" // Use hashed entity for join
        );
p
Nice!
l
Thank you for supporting 🙇