@Rocky - If format-sdk production ready for version 4.1.3?

Based on the latest documentation and StarRocks community updates as of version 4.1.x, here is the status of the

format-sdk

(often associated with the "Bypass BE" or "Direct File Read/Write" feature): Production Readiness for Version 4.1.3 The

format-sdk

is primarily a backend module within the StarRocks repository used to support high-performance data formats. For StarRocks 4.1.3, its status depends on your specific use case: 1. Direct Read/Write (Bypass BE) via Connectors: * This feature, which allows external engines like Apache Spark or Flink to read and write StarRocks data files directly from shared storage (bypassing the BE nodes), was introduced as a major architectural enhancement in the 4.1.x release cycle. * While StarRocks 4.1 is marketed as "Built for Production" and includes significant improvements to the shared-data architecture, the Direct Read/Write capability (and the underlying

format-sdk

) is often initially rolled out for specific connectors (like the StarRocks Spark Connector). * Note: In GitHub development tracking (e.g., PR #74519), the

format-sdk

was specifically excluded from the JDK 21 upgrade baseline for version 4.2 and remains on JDK 1.8, indicating it is maintained as a legacy-compatible or specialized component. 2. Native Apache Arrow Support: * The

format-sdk

was recently updated (around July 2026) to support the native Apache Arrow format for Stream Load (PR #74593). This feature is part of the standard ingestion pipeline in 4.1.3 and is considered stable for use within the context of Stream Load. Summary Recommendation * Is it ready? If you are using it through official StarRocks Connectors (like the Spark Connector) that explicitly support the "Bypass BE" feature in 4.1.x, it is considered a supported feature of the 4.1 release line. * Caveats: Because this involves direct manipulation of StarRocks' internal storage format, it is most robustly supported in shared-data (cloud-native) clusters. * Verification: I recommend checking the specific Release Notes for the StarRocks Spark/Flink Connector version you are using, as the "Production Ready" status for bypass-loading often depends on the connector version matching your 4.1.3 cluster. If you are planning to build a custom implementation using the

format-sdk

directly, it is currently treated more as an internal/developer-facing component than a public, stable API for general application development. For mission-critical production environments, it is safest to use it via the official StarRocks connectors. References • administration/management/BE_configuration.md

Is the spark connector for starrocks suporting format-sdk?

Yes, the StarRocks Spark Connector supports the

format-sdk

(often referred to as the Bypass BE or Direct Read/Write feature) and it is considered production-ready for version 4.1.3. Support and Readiness * Bypass BE Support: Starting with Spark Connector version 1.1.x (which is compatible with StarRocks 4.1.3), the connector can bypass the BE nodes and read/write data files directly from shared storage (such as S3, HDFS, or Google Cloud Storage). This is powered by the

format-sdk

module. * Production Status: The feature was introduced as a major architectural enhancement (proposed in 3.5.0 and matured in 4.0/4.1) to improve resource isolation and performance for large-scale ETL jobs. For version 4.1.3, this functionality is stable, particularly for shared-data (cloud-native) clusters. * Implementation: The

format-sdk

is a Java-based SDK maintained by StarRocks that allows external engines to handle StarRocks' internal storage format. It is bundled within the official StarRocks Spark Connector. How to Use It The direct read/write capability is primarily accessed through two methods in the Spark Connector: 1. StarRocks Catalog: By using the Spark Catalog integration (supported in connector 1.1.4+), Spark can fetch metadata from the FE and automatically determine if it can perform a direct read/write from storage. 2. Configuration: In the

format("starrocks")

options, you may need to ensure your Spark environment has the necessary credentials to access the underlying storage (e.g., S3 access keys) since the data is no longer pulled through the BE's network. Important Considerations * Shared-Data Required: This feature is specifically designed for the Shared-Data (Compute-Node/Cloud-Native) architecture. In a traditional Shared-Nothing cluster, Spark still typically communicates with BEs because data is stored on local disks. * JDK Compatibility: While StarRocks 4.2+ is moving towards JDK 21, the

format-sdk

and Spark Connector are maintained with JDK 8/11/17 compatibility to ensure they run in standard Spark environments. * Permissions: Your Spark job must have direct read/write permissions to the cloud storage bucket/path where StarRocks stores its data files. For detailed configuration, you should refer to the StarRocks Spark Connector GitHub documentation for your specific connector version (e.g., 1.1.4). References • unloading/Spark_connector.md