@Rocky starrocksFeProxySpec를 설치했어. 그리고 proxy의 service 는 LB로 설정했어. 그 이후에도 계속 307에러가 발생하는데, 따로 설정을 해줄만한 부분이 있을까?

안녕하세요! StarRocks 클러스터를 운영하시면서

starrocksFeProxySpec

설정과 LoadBalancer(LB) 사용 중 발생하는 307 에러로 인해 고민이 많으시겠군요. StarRocks에서 HTTP 307 에러는 주로 Stream Load 과정에서 발생하며, 이는 오류가 아니라 *정상적인 리다이렉션 과정*입니다. 하지만 외부에서 접속할 때 이 리다이렉션 주소가 내부 IP로 전달되어 클라이언트가 접근하지 못하는 경우가 많습니다. `starrocksFeProxySpec`을 사용하면서도 문제가 계속된다면 아래 몇 가지 사항을 확인해 보시기 바랍니다. 1. 접속 포트 확인 (8080 포트 사용) StarRocks Operator를 통해 설치된 FE Proxy는 기본적으로 두 가지 역할을 합니다. * 9030 포트: MySQL 프로토콜 프록시 (일반 쿼리용) * 8080 포트: HTTP 프록시 (Stream Load용) 만약 Stream Load를 보낼 때 FE 서비스의 기본 포트인 *8030*으로 요청을 보내고 계신다면, Proxy를 거치더라도 FE가 BE의 내부 IP로 307 리다이렉션을 보내게 되어 외부 클라이언트에서 접속이 실패할 수 있습니다. Stream Load 요청은 반드시 FE Proxy 서비스의 8080 포트로 보내야 합니다. 2. 클라이언트 설정 (

--location-trusted

) `curl`을 사용하여 Stream Load를 테스트하고 계신다면, 307 리다이렉션을 따라가기 위해 반드시

-L

또는

--location-trusted

옵션을 사용해야 합니다.

bash
curl --location-trusted -u root:password -T data.csv -H "label:123" http://<Proxy_LB_IP>:8080/api/db/table/_stream_load

FE Proxy는 FE로부터 받은 307 응답의

Location

헤더(BE 주소)를 프록시 주소로 변환해 주는 역할을 하므로, 클라이언트가 이를 신뢰하고 따라가도록 설정해야 합니다. 3. FE 설정:

enable_fe_proxy

(리다이렉션 방지) 만약 리다이렉션 구조 자체가 네트워크 환경상(방화벽 등) 복잡하여 계속 문제가 발생한다면, FE가 직접 데이터를 BE로 전달하도록 설정할 수 있습니다. 이 경우 307 리다이렉션이 발생하지 않습니다.

fe.conf

(또는 Operator의

starRocksFeSpec.config

)에 아래 설정을 추가해 보세요:

properties
enable_fe_proxy = true

* 설명: 이 설정을 `true`로 하면 FE가 HTTP 요청을 직접 받고 데이터를 BE로 포워딩합니다. 클라이언트는 BE와 직접 통신할 필요가 없어져 307 에러가 발생하지 않습니다. * 주의: 모든 데이터가 FE를 거치게 되므로 FE의 네트워크 부하가 증가할 수 있습니다. 4. 서비스 대상(Service Selector) 확인 설치하신

starrocksFeProxy

서비스가 정상적으로 FE Pod들을 바라보고 있는지 확인해 보세요.

bash
kubectl get svc -n <namespace>

`starrockscluster-sample-fe-proxy-service`와 같은 이름의 서비스가 생성되어 있고, 해당 서비스의 External-IP가 올바른지 확인이 필요합니다. 요약하자면: 1. 요청 주소를 FE 서비스(8030)가 아닌 *FE Proxy 서비스의 8080 포트*로 변경해 보세요. 2.

curl

사용 시

--location-trusted

옵션을 추가하세요. 3. 여전히 해결되지 않는다면 `fe.conf`에 `enable_fe_proxy = true`를 설정하여 리다이렉션 없이 처리하는 방법을 권장드립니다. 추가적인 로그(FE 로그나 curl 결과 상세)를 알려주시면 더 정확한 도움을 드릴 수 있습니다! 도움이 되셨길 바랍니다. References • quick_start/helm.md • administration/management/resource_management/Load_balance.md • administration/management/FE_configuration.md

@Rocky Is there a way for the FE 307 redirect to use BE pod IP instead of DNS ? We have a special setup where K8S cross-cluster networking on these IP can route anywhere.

In StarRocks, when an FE receives a request like Stream Load, it sends an HTTP 307 Temporary Redirect to the client. The

Location

header in this redirect contains the address of the selected Backend (BE). By default, the FE uses the address registered in its metadata (the Host column in

SHOW BACKENDS

). In a Kubernetes environment, the StarRocks Operator or Helm chart typically registers BE nodes using their FQDN (e.g.,

starrocks-be-0.starrocks-be-search.svc.cluster.local

) to handle pod restarts where IPs might change. If your cross-cluster networking allows routing to Pod IPs but does not support cross-cluster DNS resolution, you have a few options: 1. Use the FE Proxy (Recommended) The official solution for handling cross-network or external access issues with 307 redirects is the FE Proxy. This is an Nginx-based reverse proxy that sits in front of the FE and BEs. * How it works: Instead of the client following a 307 redirect to an unresolvable internal DNS name, the FE Proxy intercepts the traffic. It can be configured to handle the redirection or proxy the data transfer directly. * Enabling in Helm: In your

values.yaml

, enable the FE Proxy:

yaml
    starrocks:
      starrocksFeProxySpec:
        enabled: true
        service:
          type: LoadBalancer # or NodePort/ClusterIP depending on your routing

* This approach is preferred because it abstracts away the internal pod addressing. 2. Register BEs by IP (Not Recommended in K8s) You can technically force the FE to use IPs by modifying the backend registration:

sql
ALTER SYSTEM MODIFY BACKEND HOST "<be_fqdn>" TO "<be_pod_ip>";

Warning: In Kubernetes, Pod IPs are ephemeral. If a BE pod restarts and gets a new IP, the FE will lose connection to it until the metadata is updated again. The StarRocks Operator manages these lifecycle events and will typically revert them to FQDNs. 3. DNS Host Aliases / /etc/hosts If the cross-cluster networking allows IP routing, you can add

hostAliases

to the client pods (the ones initiating the Stream Load) so they can resolve the StarRocks BE FQDNs to their corresponding Pod IPs. This is a manual overhead but avoids changing the StarRocks internal configuration. Related Configurations *

enable_resolve_hostname_to_ip_in_load_error_url

(v4.0.1+): StarRocks recently introduced this BE configuration to help with debugging. While it specifically affects the

error_url

returned when a load fails, it highlights the move toward allowing IP resolution in HTTP-based responses. * FE Proxy Resolver: If using the FE Proxy, you can set a custom DNS resolver in the Helm chart to ensure it can correctly map the internal FQDNs to the routable IPs:

yaml
    starrocksFeProxySpec:
      resolver: "kube-dns.kube-system.svc.cluster.local"

Summary: The most robust way to solve this in Kubernetes without breaking the Operator's lifecycle management is to deploy the FE Proxy. This ensures that the 307 redirect doesn't leak internal FQDNs to clients that cannot resolve them. References • administration/management/BE_configuration.md • administration/management/enable_fqdn.md • faq/Deploy_faq.md • deployment/environment_configurations.md • faq/operation_maintenance_faq.md • quick_start/shared-data.md • integrations/streaming/pipe/hdfs.mdx

@Rocky The FE proxy, is it mainly used for Stream Load feature ? Can remote client directly hit FE pods and get their results without any of these 307 redirects ?

The FE Proxy is indeed primarily designed to handle the Stream Load workflow in environments like Kubernetes where clients cannot directly reach Backend (BE) pod IPs or resolve internal FQDNs. Here is the breakdown of how redirects work and how you can avoid them: 1. Is the FE Proxy only for Stream Load? While Stream Load is its most common use case (to handle the 307 redirect), the StarRocks Operator's FE Proxy (which is based on Nginx) is also used for: * Arrow Flight SQL Proxying: If you use the high-performance Arrow Flight protocol, you can enable a proxy mode where the FE acts as a bridge to BEs so the client doesn't need direct BE access. * Routine Load / Internal Tasks: In some specific cross-cluster scenarios, it helps stabilize the endpoint. 2. Can clients get results from FE pods without 307 redirects? Yes. Whether you get a 307 redirect depends entirely on the protocol and operation you are performing: *A. Standard SQL Queries (MySQL Protocol) - No Redirects* When you use a standard MySQL client or JDBC driver to connect to the FE (port 9030): * The client sends the SQL to the FE. * The FE coordinates the execution across BEs. * The BEs send the data back to the FE. * The FE sends the final result set back to the client. * There is no 307 redirect. The client only ever talks to the FE. *B. HTTP SQL API - No Redirects* If you want to use HTTP but avoid the Stream Load style redirect: * Use the HTTP SQL API (introduced in v3.2.0). * Endpoint:

POST /api/v1/catalogs/default_catalog/databases/<db>/sql

* The FE streams the results directly back to your HTTP client using chunked encoding. * There is no 307 redirect. *C. Stream Load - Redirects by Design* Stream Load is unique because it involves sending large amounts of data. To avoid bottlenecking the FE, the FE tells the client: "I've picked a BE for you, please upload the data directly to it." (HTTP 307). * If you use the FE Proxy: The client sends data to the Proxy, and the Proxy handles the "redirection" or routing internally. From the client's perspective, it just talks to one endpoint. * Without Proxy: You can bypass the FE entirely and hit a BE pod IP directly for the Stream Load:

curl --location-trusted -u root: http://<be_pod_ip>:8040/api/db/table/_stream_load

Since you are already hitting the BE, there is no redirect. Summary for your K8S setup If your goal is to avoid the 307 redirect because of DNS/IP routing issues: 1. For Queries: Just use the FE's MySQL port (9030) or the HTTP SQL API. Both keep the connection on the FE. 2. For Loading: * Option A: Use the FE Proxy. It will catch the 307 internally and keep the client's connection stable. * Option B: In your code, query

SHOW BACKENDS

to get the list of IPs, then pick one and send your Stream Load directly to that BE IP. This bypasses the FE's redirect logic entirely. Note: If you are using the StarRocks Operator, it is highly recommended to use the FE Proxy rather than manual IP routing, as the Proxy is specifically built to bridge the "Internal K8S DNS vs. External Client" gap. References • unloading/arrow_flight.md • sql-reference/http_sql_api.md • using_starrocks/async_mv/Materialized_view.md • faq/Sql_faq.md