Hello Team, - StarRocks Spark Connector: com.starrocksstarrocks spark connector 3.5 2.121.1.2 Problem: I'm trying to write a DataFrame with nested STRUCT and ARRAY types to StarRocks using the StarRocks Spark connector, but I'm getting an

UnsupportedOperationException

. Error Message: pyspark.errors.exceptions.captured.UnsupportedOperationException: Unsupported starrocks type, column name: userData, data type: struct Sample StarRocks Table DDL:

sql
CREATE TABLE test_db.test_table (
  id STRING NOT NULL,
  timestamp BIGINT NULL,
  userData STRUCT<
    name STRING,
    email STRING,
    age INT,
    address STRUCT<
      street STRING,
      city STRING
    >,
    phoneNumbers ARRAY<STRING>
  > NULL
)
PRIMARY KEY (id)
DISTRIBUTED BY HASH(id)
ORDER BY (timestamp, id);

Sample PySpark DataFrame Schema:

python
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, ArrayType

schema = StructType([
    StructField("id", StringType(), False),
    StructField("timestamp", LongType(), True),
    StructField("userData", StructType([
        StructField("name", StringType(), True),
        StructField("email", StringType(), True),
        StructField("age", IntegerType(), True),
        StructField("address", StructType([
            StructField("street", StringType(), True),
            StructField("city", StringType(), True)
        ]), True),
        StructField("phoneNumbers", ArrayType(StringType()), True)
    ]), True)
])

WriterMethod Code:

python
df.write \
    .format("starrocks") \
    .option("starrocks.fe.http.url", "<http://fe-host:8030>") \
    .option("starrocks.fe.jdbc.url", "jdbc:<mysql://fe-host:9030>") \
    .option("starrocks.table.identifier", "test_db.test_table") \
    .option("starrocks.user", "username") \
    .option("starrocks.password", "password") \
    .mode("append") \
    .save()

Questions: 1. Does the StarRocks Spark connector support writing complex types (STRUCT/ARRAY) directly? 2. If not, what's the recommended approach: - Convert STRUCT/ARRAY to JSON strings before writing? - Use a different table type (JSON instead of STRUCT)? 3. Are there plans to support complex type writes in future versions? Workaround I'm considering: Converting complex types to JSON strings using

to_json()

before writing:

python
from pyspark.sql.functions import to_json, col

df_converted = df.withColumn("userData", to_json(col("userData")))

Is this the recommended approach, or is there a better way to handle this? Thanks in advance!

Not from the starrocks team, but my team writes to iceberg and then gets starrocks to ingests from there async... complex structures work esp since we don't know when spark 4 will be supported.