http://starrocks.io logo
Join Slack
Powered by
Hello StarRocks team, I’m encountering an intermit...
# questions-and-troubleshooting
d
Hello StarRocks team, I’m encountering an intermittent issue when ingesting Parquet data from Google Cloud Storage using BROKER LOAD. The load job sometimes fails with the following error:
Copy code
```
```type: ETL_RUN_FAIL
msg: No files were found matching the pattern(s) or path(s): 
'<gs://abc/*.parquet>'. 
You should check whether there are files under the path, and make sure the process has the permission to access the path.
I am using a Google Cloud service account, and I have verified that it has the correct permissions to access this folder. The files do exist at the specified path. If I wait for a while and then run the ingest again, it usually succeeds without any changes. Could this be related to metadata caching, GCS list consistency, or some caching behavior inside the StarRocks broker layer? Please advise what could be causing this intermittent “files not found” issue.
@Kevin Cai I am using StarRocks 4.0.0, shared data architecture. Please help me check this error
k
how long is there between the file uploaded to gs and the running of the broker load in sr?
d
usually after 1 minute since the file is uploaded to gcs. But sometimes I try again after 2 hours and still get the above error.
I have many prefixes corresponding to different app data and it only happens with certain prefixes. Then I ingest again and some prefixes will have the error for up to 3 hours
k
interesting, didn't see this pattern ever before. Are you able to turn on access log for the bucket, when the not found error happens, take a look at the access log, make sure the request reaches gs server side.
d
Thank you for your assistance. I will enable the monitoring logs to check the situation and report back.
@Kevin Cai After analyzing the GCS logs, I discovered a bug. I'm ingesting concurrently from two buckets in different locations (
bucket_us
and 
bucket_us_west1
). Here is what happens: If 
bucket_us
is running and then 
bucket_us_west1
starts, the access logs show that requests are incorrectly sent to 
bucket_us
. It works fine if I run continuously on the same bucket. It also works if I wait a while before switching to a different bucket. However, when switching back and forth between different buckets, the subsequent ingest ends up using the bucket name of the previous one, even though I explicitly defined the correct bucket name in the ingestion properties
k
can you state more clearly? I think the storage volume can be bind to only one bucket and can't be changed after creation. don't understand about the switch between different location.
d
When I execute two Broker Load jobs pointing to different buckets (
bucket_us
and 
bucket_us_west1
), the second job incorrectly targets the bucket of the first job while keeping its own file path. Steps to Reproduce: 1. Job 1: Run a load from 
<gs://bucket_us>
. 2. Job 2: Immediately run a load from 
<gs://bucket_us_west1>
. Reproducible SQL: Job 1 (Targeting 
bucket_us
): 
LOAD LABEL label_bucket_us ( DATA INFILE("<gs://bucket_us/abc/*.parquet>") INTO TABLE analytics_events FORMAT AS "parquet" ) WITH BROKER ( "gcp.gcs.service_account_email" = "...", "gcp.gcs.service_account_private_key_id" = "...", "gcp.gcs.service_account_private_key" = "..." ) PROPERTIES ( "timeout" = "72000" );
Job 2 (Targeting 
bucket_us_west1
):
Copy code
LOAD LABEL label_bucket_us_west1
(
    DATA INFILE("<gs://bucket_us_west1/xyz/*.parquet>")
    INTO TABLE analytics_events
    FORMAT AS "parquet"
)
WITH BROKER
(
  "gcp.gcs.service_account_email" = "...",
  "gcp.gcs.service_account_private_key_id" = "...",
  "gcp.gcs.service_account_private_key" = "..."
)
PROPERTIES
(
    "timeout" = "72000"
);
Actual Behavior (The Bug): According to the GCS access logs, when Job 2 runs, the Broker attempts to access: 
<gs://bucket_us/xyz/*.parquet>
Expected Behavior: It should access: 
<gs://bucket_us_west1/xyz/*.parquet>
k
I see, can you open an issue on github, we will try to address this issue. sounds to me something wrong with the underlying gs filesystem cache/reuse issue.
d
I just submitted a GitHub issue for this bug: https://github.com/StarRocks/starrocks/issues/66504 . Please help check it when you have a moment. Thanks.
k
thanks!
Open in Slack
PreviousNext
Powered by