Hello! Sorry if this question has already been asked and I just missed it, but... We are building a POC with StarRocks 4.0.1, Iceberg + Nessie REST 0.108.1, Spark 3.5.1, and StarRocks Spark Connector 1.1.3. We have run into many problems when trying to read data with the StarRocks Spark Connector. For example, we have issues with struct fields, reading long string fields, and some other cases. We can see that many of these problems have already been fixed in the StarRocks Spark Connector repository, but there is still no new release. When can we expect the next release with all the latest features and fixes?

I don't see any previous messages in this conversation. Could you please describe the problem you're encountering? To help me give you the best advice, it would be useful to know: * What you were trying to do (e.g., a specific query, data loading, or configuration change). * The error message or symptoms you observed. * Your StarRocks version. * The table schema or relevant configurations, if applicable. Once you provide those details, I'll search the documentation and logs to find a solution for you!

@Rocky I'm working on a POC with the following stack: • StarRocks 4.0.1 • Spark 3.5.1 • StarRocks Spark Connector 1.1.3 • Iceberg • Nessie REST Catalog 0.108.1 The data was written from Iceberg tables to starrocks tables (using external StarRocks catalog), and I'm trying to read it through the StarRocks Spark Connector from my StarRocks tables. I'm running into several issues with complex schemas: •

STRUCT

columns are not read correctly. • Very large

STRING

columns (some of them contain large JSON documents) fail to read or are handled incorrectly. • There are also a few other schema-related issues that seem to be already reported in the connector repository. I noticed that many of these bugs appear to have already been fixed in the GitHub repository, but there hasn't been a new release for quite some time. I have a few questions: 1. Is there a recommended version combination of StarRocks and the Spark Connector that works well with tables containing complex types (

STRUCT

, nested fields, etc.)? 2. Is there any workaround for reading very large JSON strings? 3. Is there an alternative Spark connector that supports StarRocks better than the current official one? 4. Would changing StarRocks to other version help, or are these limitations entirely in the Spark Connector? 5. Is there an estimated timeline for the next Spark Connector release that includes the latest fixes?

It sounds like you're hitting some known hurdles with the Spark Connector when dealing with complex data types and high-volume string data. Here is a breakdown of clues and recommendations based on current documentation and project status: 1. Recommended Version Combination While StarRocks 4.0.1 and Spark Connector 1.1.3 is the latest "official" released combination, it's important to note that 1.1.3 was the first version to introduce

STRUCT

and nested field support for reading. Since many fixes for complex types have been merged into the repository but not yet released in a tag: * Building from source (Master branch) is the most common recommendation for users in a POC phase experiencing these issues. * Ensure you are explicitly defining the schema for complex types using the

starrocks.column.types

option. The connector often needs this metadata to serialize/deserialize

STRUCT

and

ARRAY

fields correctly. 2. Workaround for Large JSON Strings The connector relies on

jackson-core

for JSON processing. Version 1.1.3 included a fix (#129) to enlarge the maximum string length allowed by Jackson, but very large documents can still exceed Spark's internal

UTF8String

limits or cause OOMs in the connector's buffer. * Partial Extraction: If you don't need the entire JSON blob in Spark, use StarRocks'

get_json_string

or

JSON_QUERY

in a

SELECT

statement within your Spark read to extract only the necessary fields as primitive types before they hit the connector. * Flat JSON: StarRocks 4.0+ has a Flat JSON feature (enabled by default) that extracts common fields into a columnar format. Ensure your table is utilizing this so the scanner can read those fields directly as standard types rather than parsing the full JSON string. 3. Alternative Spark Connectors There isn't a "more official" alternative, but there are a few paths: * JDBC: You can use the standard Spark JDBC connector to read from StarRocks. This is much more stable for complex types and large strings but significantly slower as it funnels data through the FE rather than parallel reads from BEs. * Arrow Flight: There is ongoing work in the community for an Arrow Flight-based connector (sometimes referred to as

spark-flight-connector

), which is often more efficient for large datasets, though its support for StarRocks' specific complex type serialization may vary. 4. StarRocks Version vs. Connector Limitations Most of the issues you've described (incorrect

STRUCT

reading) are typically Connector-side logic bugs in how the scan results from the BE are mapped to Spark's InternalRow format. However, StarRocks 4.0.x specifically fixed some scan-range scheduling and "slot lookup" errors in external scans (#73225) that could affect the Spark connector. If you are on 4.0.1, you have most of these FE/BE-side fixes. The bottleneck is likely the Connector's version of the Java code. 5. Timeline for Next Release There is no publicly announced date for the next Spark Connector release (e.g., v1.1.4). Release cycles for the connectors often depend on the volume of critical bugs. * Recommendation: Check the StarRocks Spark Connector GitHub Milestones. If the fixes you need are already in the

main

branch, the safest bet for a POC is to compile the JAR yourself:

bash
    mvn clean package -DskipTests

Summary of Clues: 1. Build from Source: The fixes you saw in the repo are likely what you need. 2. Schema Declaration: Verify you are using

starrocks.column.types

. 3. JSON Flattening: Use StarRocks to pre-parse JSON into columns if the strings are too large to transfer. 4. Community Help: For a timeline update, it's best to ping the maintainers in the

#connector-spark

channel of the StarRocks Community Slack.