<@U0A71G31CDV> Upon trying to load mysql data in ...
# questions-and-troubleshooting
u
@Rocky Upon trying to load mysql data in starrocks using external catalogue with jdbc URI, I need to explicitly supply
useSSL=false
flag, otherwise insert operation in starrocks times out even though mysql (AWS rds) can support ssl data transmission. I use this query to insert.
Copy code
SUBMIT /*+ SET_VAR(query_timeout=14400) */ TASK my_big_pull AS
INSERT INTO log_fake_2
SELECT * FROM aurora_dev.test.log_fake_2;
Mysql table has 600 million rows. this issue seems to surface only for large tables; for smaller tables it works fine with or without SSL. can someone shed light what could be the issue here or could it be a bug? thanks. jdbc URI is
Copy code
jdbc:<mysql://database_url:3306?useSSL=false>
Also it works with
Copy code
jdbc:<mysql://database_url:3306?sslMode=REQUIRED&enabledTLSProtocols=TLSv1.2>
but fails with
Copy code
jdbc:<mysql://database_url:3306?sslMode=REQUIRED&enabledTLSProtocols=TLSv1.3>
so issue could be TLS 1.3. I am using starrocks 3.3.12
@Rocky would you know anything about it.
r
I'm not sure what "it" refers to! Could you please provide more details or context? Whether it's a specific StarRocks feature, an error message, or a configuration issue, I'd be happy to help once I know the topic.
This issue where
useSSL=false
resolves a timeout during large data loads is common when using the JDBC Catalog, especially with very large datasets (600M rows). Potential Causes 1. StarRocks Internal JDBC Timeouts (FE Configuration) Starting from v3.5.13, StarRocks introduced specific FE configuration parameters for JDBC catalogs that have relatively short default values: *
jdbc_query_timeout_ms
: Defaults to 30,000ms (30 seconds). This timeout is passed directly to the JDBC driver's
setQueryTimeout
. For a 600M row query, the initial execution or the time spent fetching metadata might exceed this. *
jdbc_network_timeout_ms
: Defaults to 30,000ms (30 seconds). This applies to socket reads. Why SSL matters: SSL encryption adds CPU overhead and increases the size of the packets being transmitted. This extra latency can push an operation that was "just under" 30 seconds (without SSL) over the limit, triggering the timeout. 2. MySQL JDBC Driver "Streaming" Stalls The MySQL Connector/J driver (especially 8.0.x) can occasionally hang or experience race conditions when "streaming" massive result sets. There is a known MySQL bug (#116788) where large result sets can get stuck waiting for bytes from the server. Disabling SSL simplifies the network stack and can sometimes bypass the specific code path causing the hang. 3. Missing Cursor Fetch Configuration By default, the MySQL JDBC driver may try to fetch the entire result set into memory or use a default streaming mode that is sensitive to network latency. For large tables, you should explicitly enable cursor-based fetching in the
jdbc_uri
. ────────── Recommended Solutions 1. Adjust FE Timeout Configurations If you are on StarRocks v3.5.13 or later, increase these parameters in your
fe.conf
(or via
ADMIN SET CONFIG
if mutable):
Copy code
sql
-- Increase these significantly for 600M rows
ADMIN SET CONFIG ("jdbc_query_timeout_ms" = "14400000"); -- 4 hours
ADMIN SET CONFIG ("jdbc_network_timeout_ms" = "14400000");
2. Optimize the JDBC URI Modify your JDBC URI to use cursor-based fetching. This ensures the BE nodes pull data in manageable chunks rather than trying to buffer too much or hitting network bottlenecks:
Copy code
sql
jdbc:<mysql://database_url:3306?useCursorFetch=true&defaultFetchSize=5000&useSSL=true>
Note: Using
useCursorFetch=true
requires
defaultFetchSize
to be set to a positive integer.
3. Use the correct Hint In recent StarRocks versions,
INSERT
operations are controlled by
insert_timeout
rather than
query_timeout
. Try updating your hint:
Copy code
sql
SUBMIT /*+ SET_VAR(insert_timeout=14400) */ TASK my_big_pull AS
INSERT INTO log_fake_2
SELECT * FROM aurora_dev.test.log_fake_2;
4. Check BE Logs If the task still fails, check the
<http://be.INFO|be.INFO>
or
be.WARNING
logs on the BE nodes. Look for exceptions like
Communications link failure
,
SocketTimeoutException
, or
Broken pipe
. This will confirm if the timeout is happening at the socket level or if the JDBC driver itself is throwing an error. If you cannot increase the FE timeouts (e.g., you are on an older version), keeping
useSSL=false
is a valid workaround provided your network between StarRocks and AWS RDS is secure (e.g., within the same VPC or via VPC Peered over a private link). Referencesadministration/user_privs/ssl_authentication.mddata_source/catalog/jdbc_catalog.mddata_source/catalog/elasticsearch_catalog.md
The behavior you're seeing—where TLS 1.2 works for large 600M row transfers but TLS 1.3 causes a timeout—is a known issue often related to the JDK version running on your StarRocks Backend (BE) nodes. In StarRocks 3.3.x, JDBC catalog queries are executed by a Java process (the JDBC Bridge) on the BE nodes. Here is why TLS 1.3 is likely failing for your large data load: 1. JDK 11 TLS 1.3 Implementation Bugs StarRocks 3.3.x typically runs on JDK 11. Early versions of JDK 11 (and even some later updates) have documented performance issues and bugs with TLS 1.3, including: * CPU Spikes: TLS 1.3 can consume significantly more CPU (up to 3x in some Java 11 environments) compared to TLS 1.2 during large data streaming. * Handshake/Stall Bugs: Known JDK bugs (like
JDK-8214418
or
JDK-8213202
) can cause the connection to hang or enter a 100% CPU loop when processing massive result sets over TLS 1.3. * Packet Handling: TLS 1.3 changed how "half-close" and session resumptions work, which sometimes conflicts with how the MySQL JDBC driver "streams" data for very large tables. 2. AWS Aurora / RDS Cipher Compatibility AWS Aurora MySQL (especially versions compatible with 8.0) supports TLS 1.3, but it uses a restricted set of ciphers. If there is a mismatch or a slow negotiation between the BE's JVM and the RDS instance, the initial handshake might succeed, but the subsequent heavy data stream can trigger a timeout if the BE cannot process the encrypted packets fast enough. ────────── Recommended Solutions Option A: Stick with TLS 1.2 (Recommended) Since TLS 1.2 is secure and works perfectly for your 600M row dataset, the most stable fix is to explicitly lock the protocol in your JDBC URI:
Copy code
sql
jdbc:<mysql://database_url:3306?enabledTLSProtocols=TLSv1.2>
Option B: Check and Upgrade the JDK on BE Nodes Check the Java version used to start your BE nodes. If it is an older JDK 11 version (e.g., 11.0.2 or earlier), upgrade to JDK 11.0.15+ or JDK 17. * In
$BE_HOME/bin/start_be.sh
, ensure
JAVA_HOME
points to a modern JDK. _ _Note: StarRocks 3.3 supports JDK 11 or later.* Option C: Adjust JDBC Bridge Memory/Timeouts Large data loads over SSL require more memory for buffer management. You can increase the memory allocated to the BE's JDBC bridge by setting the following in your
be.conf
and restarting the BE:
Copy code
bash
# Increase the heap size for the JDBC Bridge (default is often small)
export JAVA_OPTS="-Xmx4g"
Option D: Enable Streaming/Cursor Fetch For 600M rows, the MySQL driver must stream data rather than loading it into memory. Ensure your URI also uses these parameters to prevent the driver from stalling:
Copy code
sql
jdbc:<mysql://database_url:3306?useCursorFetch=true&defaultFetchSize=5000&enabledTLSProtocols=TLSv1.2>
Summary The issue is likely not a bug in StarRocks itself, but a performance bottleneck or bug in the Java 11 TLS 1.3 stack when handling high-throughput encrypted streams. For production workloads of this scale, TLS 1.2 is currently more stable for JDBC-based ingestion in StarRocks 3.3. Referencesadministration/management/FE_configuration.mdrelease_notes/release-3.3.mdadministration/user_privs/ssl_authentication.md