@Rocky, is it possible that I can’t use the transactional Stream Load interface because I’m running the FE and BE on the same machine via Docker Compose?

Running it via Docker Compose on a single machine does not inherently prevent you from using the transactional Stream Load interface, but it often causes networking/connectivity issues due to how StarRocks handles load requests. The common reason you might find it "impossible" or failing in this setup is the HTTP Redirect mechanism: 1. The Redirect Problem: When you send a

/api/transaction/load

request to the FE (usually port 8030), the FE selects a BE to handle the data and sends an HTTP 307 redirect back to your client. 2. Internal Hostnames: Inside Docker Compose, the BE typically registers itself with the FE using its internal container name (e.g.,

starrocks-be

) or internal IP (e.g.,

172.x.x.x

). 3. Client Connection Failure: If your client (e.g.,

curl

or a Python script) is running on your host machine (outside the Docker network), it will receive a redirect to

<http://starrocks-be:8040/>...

. Since your host machine doesn't know what

starrocks-be

is, the connection fails. How to make it work: * Map BE Ports: Ensure you have mapped the BE's HTTP port in your

docker-compose.yml

(default

8040

).

yaml
    ports:
      • "8040:8040"

* Update Hosts File: Add an entry to your host machine's

/etc/hosts

(Linux/Mac) or

C:\Windows\System32\drivers\etc\hosts

(Windows) to map the BE's container name to

127.0.0.1

.

text
    127.0.0.1 starrocks-be

* Use `--location-trusted`: If using

curl

, you must use the

--location-trusted

flag. This allows

curl

to follow the redirect and pass your credentials to the BE node. * Single Node Setup: If you are using the "all-in-one" Docker image (where FE and BE are in the same container), you usually won't hit this as long as

8030

and

8040

are both mapped to

localhost

. Other Considerations: * Version Support: Ensure you are running v2.4 or later, as the transactional interface (2PC) was introduced in that version. * Resource Constraints: Running FE and BE on one machine via Docker can be memory-intensive. Ensure Docker has enough RAM (at least 4GB-8GB) assigned to it, or the BE process might crash during a heavy load. If you are seeing a specific error message (like

Connection refused

or

Could not resolve host

), it is almost certainly the redirect/hostname issue mentioned above. References • loading/Stream_Load_transaction_interface.md • administration/management/monitoring/metrics.md

@Rocky we are usuing the 4.1.1 version

Yes, you can absolutely use the transactional Stream Load interface with StarRocks v4.1.1 on a single machine via Docker Compose. In fact, version 4.1.x is highly recommended for container environments as it fixed several stability issues found in earlier 4.0.x and 4.1.0 releases. The reason it might seem like it's not working is almost always a networking/connectivity issue caused by how Docker handles internal hostnames and how StarRocks handles load redirects. Why it might be failing in Docker: When you call the Stream Load transaction API (

/api/transaction/load

), the FE (Front-End) receives the request and sends an HTTP 307 Redirect to the BE (Back-End) that will actually process the data. * Internal Hostnames: Inside Docker, the BE registers itself with the FE using its container name (e.g.,

starrocks-be

) or its internal container IP. * The Redirect Loop: If your client (curl, Python, etc.) is running on your host machine (outside the Docker network), it will receive a redirect to

<http://starrocks-be:8040>

. Your host machine doesn't know what

starrocks-be

is, so the connection fails. Steps to fix this in Docker Compose: 1. Expose the BE Port: In your

docker-compose.yml

, ensure the BE HTTP port (default

8040

) is mapped to the host.

yaml
    services:
      starrocks-be:
        ports:
          • "8040:8040"
          # ... other ports

2. Edit your Host file: Map the BE's internal name to

127.0.0.1

on your local machine so your client can follow the redirect. * Linux/Mac:

/etc/hosts

* Windows:

C:\Windows\System32\drivers\etc\hosts

* Add line:

127.0.0.1 starrocks-be

(replace

starrocks-be

with whatever your service name is in Docker Compose). 3. Use `--location-trusted`: If using

curl

, you must use this flag. Standard

--location

(or

-L

) will follow the redirect but will not resend your authentication headers to the BE, causing a

401 Unauthorized

error.

bash
    curl --location-trusted -u root: -H "label:job_1" \
         -XPOST <http://localhost:8030/api/transaction/begin>

Specific to v4.1.1: * Multi-table Support: Since you are on v4.1.1, you can take advantage of Multi-table Transactions. You can call

/api/transaction/load

multiple times for different tables within the same transaction. * Stability: v4.1.1 includes fixes for a "PREPARING forever" bug that occurred in some earlier versions of the transaction interface. Recommendation: If you are getting a

Connection Refused

or

Could not resolve host

error, check your networking as described above. If you are getting a specific StarRocks error message (like "Transaction Not Found"), please share the specific error code. References • loading/Stream_Load_transaction_interface.md • administration/management/FE_configuration.md • loading/Loading_intro.md References • loading/Stream_Load_transaction_interface.md • loading/Loading_intro.md

@Rocky i am getting: err: { “type”: “Error”, “message”: “Transaction load rejected (table=products_stats, label=cdc-1784141508667-64089f7a): [FAILED] class com.starrocks.common.MetaNotFoundException: Failed to find table products_stats in db treinta”, “stack”: Error: Transaction load rejected (table=products_stats, label=cdc-1784141508667-64089f7a): [FAILED] class com.starrocks.common.MetaNotFoundException: Failed to find table products_stats in db treinta at StarRocksService.transactionLoad (/Users/taity/Work/sanbox/treinta-cdc-starrocks-worker/src/starrocks/starrocks.service.ts31113) at processTicksAndRejections (nodeinternal/process/task queues95:5) at StarRocksService.processTable (/Users/taity/Work/sanbox/treinta-cdc-starrocks-worker/src/starrocks/starrocks.service.ts20322) at StarRocksService.upsertBatch (/Users/taity/Work/sanbox/treinta-cdc-starrocks-worker/src/starrocks/starrocks.service.ts13824) at ProcessorService.processBatch (/Users/taity/Work/sanbox/treinta-cdc-starrocks-worker/src/processor/processor.service.ts17726) at CdcSqsHandlerService.doFlush (/Users/taity/Work/sanbox/treinta-cdc-starrocks-worker/src/sqs/cdc-sqs-handler.service.ts11520) }

The error

MetaNotFoundException: Failed to find table products_stats in db treinta

is an explicit message from StarRocks stating that it cannot locate the table or the database you are targeting. This is usually not caused by Docker itself, but by a mismatch in the request parameters or the state of your database. Here are the most likely causes and how to fix them: 1. Case Sensitivity By default, StarRocks table and database names on Linux are case-sensitive. * Check if your table is actually named

Products_Stats

or

products_stats

in the database. * Run

SHOW TABLES IN treinta;

in your SQL editor to verify the exact spelling and casing. * In your TypeScript code, ensure the headers

db

and

table

match this exactly. 2. Missing Database/Table Verify that the database

treinta

actually exists and contains the table

products_stats

.

sql
-- Check if database exists
SHOW DATABASES LIKE 'treinta';

-- Check if table exists
USE treinta;
SHOW TABLES LIKE 'products_stats';

3. Header/Parameter Mismatch in TypeScript In the Transactional Stream Load API, you must provide the database and table names in the headers of the

/api/transaction/load

request. Based on your stack trace, check your

StarRocksService.transactionLoad

implementation: Ensure you are sending these headers: * `db`:

treinta

* `table`:

products_stats

* `label`: (your unique label) Example of what the underlying request should look like:

http
PUT /api/transaction/load HTTP/1.1
Host: <be_host>:8040
db: treinta
table: products_stats
label: cdc-1784141508667-64089f7a
...

4. Multi-Table Transaction Requirement Since you are on v4.1.1, if you are loading into multiple tables within the same transaction, you must include the header

transaction_type: multi

in all calls:

begin

,

load

, and

commit

. If you started the transaction (

/api/transaction/begin

) without the

multi

header, StarRocks expects the table to be defined at the "begin" stage and might reject different table names during the "load" stage. 5. Permissions Ensure the user configured in your

StarRocksService

has the

INSERT

privilege on that specific table.

sql
GRANT INSERT ON TABLE treinta.products_stats TO 'your_user'@'%';

6. Search Path / Fully Qualified Names In the Stream Load API, StarRocks does not use a "current database" context from a session like a JDBC connection does. You must explicitly provide the

db

header in every request. If your code is omitting the

db

header or sending an empty string, it will fail to find the table. Recommended Troubleshooting Step: Try to run a manual

curl

command from your terminal to see if it works outside of your Node.js code. This will help isolate if the issue is in your code's request logic or the StarRocks setup:

bash
curl --location-trusted -u root: -H "label:test_label_1" \
     -H "db:treinta" -H "table:products_stats" \
     -XPOST <http://localhost:8030/api/transaction/begin>

If this

begin

call fails with the same error, the table/db definitely does not exist as specified. References • loading/Stream_Load_transaction_interface.md • sql-reference/sql-statements/data-definition/CREATE_TABLE.md • administration/management/FE_configuration.md • loading/Loading_intro.md References • loading/Stream_Load_transaction_interface.md