http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> ```CoreDNS High CPU — StarRocks Sha...
# questions-and-troubleshooting
k
@Rocky
Copy code
CoreDNS High CPU — StarRocks Shared-Data DNS Resolution Issue

  Environment:
  - Kubernetes (EKS)
  - StarRocks shared-data cluster with CN nodes writing to S3
  - CoreDNS v1.13.2, 2 replicas

  ---
  Observation:

  CoreDNS pods hitting 100%+ CPU during peak write hours. Pattern exactly matches Lake Memtable Flush Throughput graph — when flush rate increases, CoreDNS CPU
  increases proportionally.

  ---
  Root Cause:

  Every S3 flush from StarRocks CN nodes generates 10 DNS queries instead of 1-2 due to:

  1. Kubernetes default ndots:5 — for <http://starrocks-s3-bucket.s3.ap-south-1.amazonaws.com|starrocks-s3-bucket.s3.ap-south-1.amazonaws.com> (4 dots < 5), the OS resolver tries 4 search domain combinations before resolving the
   real name:
  *.s3.ap-south-1.amazonaws.com.cluster.local     → NXDOMAIN
  *.s3.ap-south-1.amazonaws.com.svc.cluster.local → NXDOMAIN
  *.s3.ap-south-1.amazonaws.com.<namespace>.svc.cluster.local → NXDOMAIN
  *.s3.ap-south-1.amazonaws.com.compute.internal  → NXDOMAIN
  *.<http://s3.ap-south-1.amazonaws.com|s3.ap-south-1.amazonaws.com>                   → NOERROR ✅

  2. Both A and AAAA queries — doubles the queries (IPv4 + IPv6 lookup per attempt)
  3. No DNS caching in StarRocks CN — CN nodes use AWS SDK for C++ which relies on glibc's getaddrinfo(). Unlike Java SDK (used by SeaTunnel etc.), glibc has zero DNS
  caching by default. Every new S3 connection resolves DNS fresh.

  Result: At peak flush rate of 3-4/s per CN × 6 CN nodes × 10 DNS queries = ~500 DNS queries/second saturating CoreDNS.

  ---
  Ask for StarRocks team:

  1. Does StarRocks CN/BE have any configuration to cache DNS resolutions? Similar to Java's networkaddress.cache.ttl — a way to cache resolved S3 endpoint IPs at the
  application level.
  2. Does StarRocks reuse S3 connections between flushes? If S3 HTTP connections are reused (connection pool), DNS would only be resolved once per connection lifetime
  rather than per flush.
  3. Is there a plan to support dnsConfig in the StarRocks Kubernetes operator? PR #696 (feat: add dnsConfig and dnsPolicy support) would allow setting ndots: 2
  directly in the StarRocksCluster CRD — this would be the cleanest Kubernetes-native fix
r
Regarding your observation of high CoreDNS CPU usage during StarRocks shared-data S3 flushes, here are the answers based on current StarRocks documentation and repository status: 1. Does StarRocks CN/BE have any configuration to cache DNS resolutions? No. StarRocks CN and BE nodes are written in C++ and primarily use the AWS SDK for C++, which relies on the standard library’s 
getaddrinfo()
. Unlike the Java SDK, glibc does not provide application-level DNS caching by default. • Recommendation: The standard practice for this scenario is to deploy a node-level DNS cache such as NodeLocal DNSCache in your Kubernetes cluster. This reduces the load on CoreDNS by caching lookups on each node and handles the 
ndots
search path overhead locally. 2. Does StarRocks reuse S3 connections between flushes? StarRocks uses a client cache for S3 (and other object stores) to avoid the overhead of re-initializing the SDK client for every request. However, the level of connection reuse (keep-alive) depends on the underlying 
HttpClient
configuration in the C++ SDK. • While the client object is reused, the frequent "flush" pattern in shared-data mode can still trigger new name resolutions if the connections are not kept alive long enough or if the SDK’s internal connection pool exhausts its idle connections. • There are currently no user-facing 
be.conf
parameters to tune the specific keep-alive or connection pool size for the S3-specific 
HttpClient
beyond the internal defaults. 3. Plan for 
dnsConfig
support in the StarRocks K8s Operator Yes, there is an active PR for this. • PR #696 (feat: add dnsConfig and dnsPolicy support) in the 
starrocks-kubernetes-operator
repository is specifically designed to address this. It allows you to set 
dnsPolicy
and 
dnsConfig
(including 
ndots: 2
) directly in the 
StarRocksCluster
CRD for FE, BE, and CN components. • Status: As of early 2026, this feature is in the final stages of review/integration. If you are using a version where this is not yet merged, you can temporarily work around this by: 1. Using a Mutating Admission Webhook to inject the 
dnsConfig
into StarRocks pods. 2. Appending a dot to your S3 endpoint (e.g., 
s3.ap-south-1.amazonaws.com.
) in your storage volume configuration to force it to be treated as an FQDN, bypassing the 
ndots
search. Summary Checklist for Mitigation 1. Enable NodeLocal DNSCache: This is the most effective way to handle the 500+ QPS without saturating CoreDNS. 2. FQDN Suffix: Try using 
bucket-name.s3.region.amazonaws.com.
(with the trailing dot) in your StarRocks configuration. 3. Monitor PR #696: Once merged, you can apply the following to your CN spec:
Copy code
yaml
   spec:
     starRocksCnSpec:
       dnsConfig:
         options:
           • name: ndots
             value: "2"
References • deployment/shared_data/s3.md • administration/management/BE_configuration.md • quick_start/shared-data.md • administration/management/FE_configuration.md
k
@Rocky why is the dns not getting cached on starrocks end?
r
The lack of DNS caching on the StarRocks CN/BE side is primarily a result of the C++ architecture and its reliance on the standard Linux system libraries. Here is the technical breakdown: 1. C++ and Glibc Statelessness Unlike the Java-based FE (which uses the JVM's 
networkaddress.cache.ttl
), the CN and BE nodes are written in C++. They resolve hostnames using the standard 
getaddrinfo()
function from glibc. • The Issue: By design, 
glibc
is stateless and does not cache DNS lookups. Every time the AWS C++ SDK needs to resolve an S3 endpoint, it makes a new request to the OS resolver, which in Kubernetes means a network trip to CoreDNS. • Why it's not in the app: Many high-performance C++ applications avoid internal DNS caching to prevent "stale entry" bugs (where the app keeps trying to connect to a retired IP). They typically delegate caching to the operating system level. 2. AWS C++ SDK Behavior StarRocks uses the AWS SDK for C++ for S3 communication. • The SDK manages a pool of HTTP connections, but it does not implement its own DNS cache layer. • If a connection is closed or a new one is needed (which happens frequently during high-throughput "Lake Memtable" flushes), the SDK triggers a fresh 
getaddrinfo()
call. 3. Contrast with StarRocks FE (Java) Interestingly, StarRocks recently introduced a DNS cache for the FE only (starting in v3.5.11 and v4.0.4) via the 
dns_cache_ttl_seconds
parameter. This works because it is simply a wrapper around the JVM's internal caching mechanism. There is currently no equivalent C++ implementation inside the CN/BE. Why this is a "Perfect Storm" in Kubernetes The issue is amplified specifically by the Kubernetes environment: • ndots: 5: Because the S3 URL (e.g., 
<http://s3.ap-south-1.amazonaws.com|s3.ap-south-1.amazonaws.com>
) has 4 dots, Kubernetes forces the resolver to try 4 search-path combinations (like 
...svc.cluster.local
) before trying the actual FQDN. • Result: Every single "flush" that triggers a resolution results in 10 queries (5 for A records, 5 for AAAA records) being sent to CoreDNS. How to Fix It Since there is no "enable_dns_cache" setting in 
be.conf
, you should use the standard Kubernetes solutions for high-performance C++ apps: 1. Use Trailing Dots (The "Zero-Config" Fix): Update your S3 endpoint in the Storage Volume configuration to include a trailing dot: 
s3.ap-south-1.amazonaws.com.
(note the final 
.
). This tells the OS it is a Fully Qualified Domain Name (FQDN) and skips the 4 unnecessary search-path lookups. 2. NodeLocal DNSCache: This is the recommended production fix for EKS. It runs a lightweight DNS caching agent on every node, intercepting these lookups before they ever hit the CoreDNS service. 3. Wait for Operator PR #696: This will allow you to set 
ndots: 2
in the pod spec, which would also prevent the search-path "amplification" effect. References • deployment/shared_data/s3.md • quick_start/shared-data.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • integrations/streaming/pipe/s3.mdx
Open in Slack
PreviousNext
Powered by