Bit of a random question.. I’m wondering if there ...
# random
m
Bit of a random question.. I’m wondering if there is anything available in Flink or PyFlink where I define a schema, and then provide a JSON string or object/dict, and Flink/PyFlink does the work to produce a schema-conforming object for storage, e.g. set missing fields to None, drop fields that are not in the schema, and performing basic type coercion e.g. string -> int
so I’ve been looking into ways to avoid repeating schemas so often. I know there are different schema layers with their own data types (SQL, Table, DataStreams)…
Copy code
from pyflink.table.types import _to_java_type
x = table.get_schema().to_row_data_type()
_to_java_type(x)
results in
Copy code
TypeError: The length limit must be 0x7fffffff(2147483647) for VarCharType, got VarCharType(36, true)
I wonder if there’s a more forgiving conversion function available?
This same error appears again when trying this
Copy code
from pyflink.table.descriptors import Json

Json(schema=table.get_schema().to_row_data_type())