http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> i'm getting `dat does not exist` on...
# questions-and-troubleshooting
g
@Rocky i'm getting 
dat does not exist
on a query, any clue why? or what i can look into?
r
The error 
dat does not exist
(which is likely a fragment of an error like 
database 'dat' does not exist
, 
table 'dat' does not exist
, or 
catalog 'dat' does not exist
) typically means that the StarRocks engine cannot find the object you are referencing in your query. Here are the most common reasons and things you can look into: 1. Check your Search Path (Catalog and Database) If you are running a query like 
SELECT * FROM dat
, StarRocks looks for a table named 
dat
in your current database and catalog. * Verify Catalog: Run 
SELECT current_catalog();
. If you are trying to query an external table (like Hive or Iceberg), ensure you have switched to the correct catalog using 
SET CATALOG <catalog_name>;
. * Verify Database: Run 
SELECT database();
. If it returns NULL or a different database, the table won't be found. Use 
USE <database_name>;
to switch. * Use Fully Qualified Names: Try using the full path: 
catalog_name.database_name.table_name
. 2. Check for Typos and Case Sensitivity * StarRocks identifiers (database and table names) are generally case-sensitive depending on your 
lower_case_table_names
configuration (which defaults to being sensitive on Linux). * Run 
SHOW DATABASES;
or 
SHOW TABLES;
to confirm the exact spelling and casing of the object. 3. Check Permissions Even if the table exists, if your user does not have 
SELECT
or 
USAGE
privileges on the database/table, StarRocks might return an error indicating it doesn't exist or is inaccessible. * Check your privileges with: 
SHOW GRANTS FOR CURRENT_USER;
. 4. Querying External Data (Data Lake) If 
dat
is an external catalog or a table in a data lake: * Catalog existence: Run 
SHOW CATALOGS;
to make sure the catalog was created successfully. * Metadata Refresh: If the table was recently created in an external system (like Hive or Glue), StarRocks might not have cached the metadata yet. Try refreshing the metadata:
Copy code
sql
    REFRESH EXTERNAL TABLE [database_name.]table_name;
    -- Or for the whole catalog
    ALTER CATALOG <catalog_name> SET ("metadata_refresh_interval_secs" = "0");
5. Using the 
FILES()
Table Function If you are using the 
FILES()
function to query data directly from S3/HDFS and see a "does not exist" error: * Double-check the path and credentials. _ Ensure the file pattern (e.g., 
_.parquet
or 
*.dat
) matches actual files in the storage bucket. What to check next: Could you provide the full query you are running and the exact error message? That would help narrow down if it's a catalog, database, or a specific file path issue. References • faq/resource_isolation_faq.md • faq/loading/Loading_faq.md • loading/loading_introduction/troubleshooting_loading.md • quick_start/shared-data.md • faq/loading/Broker_load_faq.md • loading/Etl_in_loading.md
g
@Rocky i'm getting Common causes for this error are: 1. A schema change has been made to the table 2. The current query is a slow query For the first case, you can retry the current query. For the second case, see "admin show frontend config like 'lake_autovacuum_grace_period_minutes'" for reference.: BE:1255144 but i haven't made any changes and the query isn't slow
r
The error code BE:1255144 typically occurs in StarRocks Shared Data (cloud-native) clusters. It indicates that the specific data version the query is trying to access is no longer available in the metadata service. Even if you haven't performed manual schema changes and the query doesn't feel "slow," this error is usually triggered by the background AutoVacuum process cleaning up older data versions too aggressively. Why this happens In a shared-data architecture, StarRocks uses MVCC (Multi-Version Concurrency Control). When data is loaded or compacted, new versions are created. The AutoVacuum process automatically deletes old, obsolete versions after a certain "grace period." If a query starts and picks version N, but before it finishes (or even before it starts execution after planning), the AutoVacuum process determines that version N is old enough to be deleted, the query will fail with this error because the underlying files or metadata are gone. Common causes (even for fast queries) 1. High Frequency of Updates/Compactions: If your cluster is under heavy load or performing very frequent data ingestion, many versions are generated quickly. 2. Short Grace Period: The default 
lake_autovacuum_grace_period_minutes
might be too short for your environment (the default was 5 minutes in older 3.x versions, and increased to 30 minutes in newer ones). 3. Clock Skew: If there is a time difference between your FE (Frontend) nodes, the vacuum logic (which is time-based) might inadvertently delete versions that are still technically within the grace period relative to another node. How to Fix 1. Increase the Grace Period The most common fix is to increase the time historical versions are retained. We recommend setting this to at least 60 minutes (or longer if you have very long-running jobs). Run this command on your FE:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");
2. Check Node Time Sync Ensure that all FE and CN/BE nodes have their clocks synchronized (e.g., using 
ntp
or 
chrony
). Significant clock skew is a known cause for premature metadata deletion in shared-data clusters. 3. Monitor Compaction and Versioning If the error persists, check if a specific table is undergoing extreme churn: * Check the number of versions: 
SHOW PROC '/compactions';
* Verify the current config: 
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum%';
Note: If you are seeing this frequently immediately after a cluster restart or during high FE leader switching, it might be a transient metadata sync issue, but increasing the grace period is still the primary recommendation. References • faq/operation_maintenance_faq.md • quick_start/shared-data.md • deployment/shared_data/s3.md • faq/resource_isolation_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky will increasing the grace period help me get out of the current jam?
r
Whether "increasing the grace period" will help depends entirely on the type of "jam" you are facing. In StarRocks, there are three main types of "grace periods" typically discussed. Depending on your situation, increasing it could either be the solution or it could make the problem worse. 1. The "Data Version Not Found" Jam (Shared-data Clusters) If you are running a shared-data (lake-based) cluster and your long-running queries are failing with errors like "version not found" or "metadata not found," then YES, increasing the grace period will help. * Parameter: 
lake_autovacuum_grace_period_minutes
(FE configuration) * Default: 30 minutes (in recent versions). * The Problem: StarRocks automatically cleans up old data versions after compaction. If a query takes 45 minutes to run, but the cleanup (vacuum) happens at 30 minutes, the query will fail because its data version was deleted while it was still reading. * The Fix: Increase this value to be longer than your longest-running query (e.g., set it to 
60
or 
120
). 2. The "Disk Full / Storage Quota" Jam If your cluster is in a jam because disks are full or you are exceeding your object storage budget, then NO, increasing the grace period will make it worse. * Parameters: 
lake_autovacuum_grace_period_minutes
(Shared-data) or 
trash_file_expire_time_sec
(Shared-nothing BE trash). * The Problem: These settings determine how long deleted data or old versions sit in "trash" before being permanently deleted. * The Solution: You actually want to decrease these values to force StarRocks to clean up space more aggressively. You can also manually clear the 
storage/trash
directory on BE nodes if you are in an emergency. 3. The "Queries Failing During Upgrade" Jam (Graceful Exit) If you are in a jam where queries are being killed every time you restart a node for an upgrade or maintenance, then YES, increasing the timeout will help. * Parameter: 
--timeout
flag in 
stop_be.sh -g
or 
stop_fe.sh -g
. * The Problem: By default, a "Graceful Exit" waits for a set period (often 60 seconds) for queries to finish before force-killing the process. If your queries take longer, they get aborted. * The Fix: Increase the 
--timeout
value when running the stop script (e.g., 
./stop_be.sh -g --timeout 300
) to give your queries more time to drain naturally. 4. The "Accidental Drop" Jam If you accidentally dropped a table or partition and need to get it back: * Parameter: 
catalog_trash_expire_second
(FE configuration, default 86400s / 1 day). * The Fix: Increasing this won't bring back data already gone, but it "buys you time" for other tables if you're worried about them being purged. Use the 
RECOVER
command immediately to restore the dropped metadata. Which "jam" are you currently in? If you can provide the specific error message or symptom, I can give you the exact parameter to tweak. References • administration/management/graceful_exit.md • administration/management/FE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md
g
@Rocky if i update 
lake_autovacuum_grace_period_minutes
from 30 to 60 how does that make the next query work?
r
Hi! I'm Rocky. Updating 
lake_autovacuum_grace_period_minutes
from 30 to 60 primarily affects how StarRocks handles long-running queries in a shared-data (StarRocks 3.x Lake) architecture. Here is how it makes your next query work: 1. Longer Query Safety Window This parameter defines how long StarRocks retains historical data versions after they have been compacted. * At 30 minutes: If a query starts and takes longer than 30 minutes, the AutoVacuum process might delete the old data files that the query was originally reading, causing the query to fail with an error (typically related to "file not found" or metadata inconsistencies). * At 60 minutes: You are doubling that "grace period." This ensures that any query starting now can run for up to 60 minutes without the risk of its underlying data being cleaned up by the background garbage collection process. 2. Prevention of "File Not Found" Errors If you were previously experiencing query failures on very large scans or complex joins that exceeded 30 minutes, increasing this value to 60 will allow those queries to complete successfully. It ensures the data consistency of the snapshot the query is using for a longer duration. 3. Impact on Storage Keep in mind that by increasing this value, you are instructing StarRocks to keep obsolete data versions in your object storage (like S3 or OSS) for an extra 30 minutes before they are physically deleted. This will result in a temporary increase in storage usage, as "garbage" data is held longer to satisfy the longer grace period. Summary: If your "next query" is a long one, changing this setting gives it a larger safety buffer, ensuring the data it needs isn't deleted while the query is still in progress. You should set this value to be greater than your longest-running query's duration. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
g
@Rocky i am getting this error immediately though
r
To give you a precise solution, could you please provide: 1. The exact error message you are seeing. 2. The SQL query or command you were trying to run. Errors that happen "immediately" are often related to: * Syntax Errors (1064): A typo or use of a reserved keyword. * Connection/Auth Issues (1045/1130): Incorrect username, password, or IP permissions. * Metadata Issues (1049/1051/1054): The database, table, or column doesn't exist. * Cluster State (5021/1064): No surviving BE nodes or the cluster is still starting up. Once you provide the error text, I can pinpoint exactly what's going wrong! References • sql-reference/Error_code.md • faq/Sql_faq.md • sql-reference/sql-statements/keywords.md
g
@Rocky I'm getting SQL Error [1064] [42000]: load_segments failed tablet:1230741 rowset:23 segid0 starlet err Object myblob.dat does not exist: Common causes for this error are: 1. A schema change has been made to the table 2. The current query is a slow query For the first case, you can retry the current query. For the second case, see "admin show frontend config like 'lake_autovacuum_grace_period_minutes'" for reference.: BE:1255414
r
Hi, I'm Rocky! This error usually occurs in StarRocks Shared-data (Cloud-native) clusters. The error 
load_segments failed ... Object <myblob>.dat does not exist
means that the query tried to read a data file that has already been deleted from the object storage by StarRocks' background garbage collection process (AutoVacuum). Why did this happen? In a shared-data cluster, when data is compacted, StarRocks creates new data files and marks the old ones for deletion. The parameter 
lake_autovacuum_grace_period_minutes
defines how long these old files are kept before they are permanently deleted. If a query runs longer than this grace period, the files it was originally reading might be cleaned up before the query finishes, leading to the "Object does not exist" error. How to fix it: 1. Check your current configuration Run the following command to see your current grace period (the default is often 5 or 30 minutes depending on your version):
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';
2. Increase the grace period You should set this value to be longer than your longest-running query. For example, if you have queries that take 45 minutes, you should set this to at least 60 minutes:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");
3. Concurrent Schema Changes If you weren't running a slow query but were performing a Schema Change (like adding a column or changing a data type) while querying the same table, the query might have been looking for data segments that were invalidated by the schema change. In this case, simply retrying the query should work. Summary of the parameter: * Parameter: 
lake_autovacuum_grace_period_minutes
* Default: 30 (in recent versions like v3.1.10+, v3.2.5+, v3.3.0+) * Recommended Action: Set it to a value greater than the maximum expected execution time of your queries. Let me know if you need help adjusting this or investigating further! References • faq/resource_isolation_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
g
@Rocky I get that error immediately running a query like SELECT _raw_batches_._event_day_, *ARRAY_AGG*(_raw_batches_.raw_import_timestamp) as _raw_import_timestamps_ FROM ( SELECT rb.raw_import_timestamp, *date*(rb.
event_time
) AS _event_day_ FROM 
events
.
table
AS rb WHERE 1=1 AND rb.raw_import_timestamp < *DATE_ADD*(*NOW*(), INTERVAL -2 MINUTE) -- wait 2 minutes after a batch has loaded just in case another PIPE thread has not loaded yet. AND *DATE_TRUNC*('day', *CAST*(
event_time
 AS DATETIME)) >= *DATE*(*DATE_SUB*(*CURDATE*(), INTERVAL 15 DAY)) GROUP BY rb.raw_import_timestamp, *date*(rb.
event_time
) ) AS _raw_batches_ LEFT OUTER JOIN ( SELECT -- ALL C Batches already imported cb.raw_import_timestamp, -- RAW Import timestamp batch, *DATE*(*CAST*(cb.
event_time
AS datetime)) AS _event_day_ FROM 
events
.
table2
AS cb WHERE event_day >= *DATE*(*DATE_SUB*(*CURDATE*(), INTERVAL 15 DAY)) GROUP BY cb.raw_import_timestamp, *DATE*(*CAST*(cb.
event_time
AS datetime)) ) AS _c_batches_ -- The batch information matches ON _raw_batches_.raw_import_timestamp = _c_batches_.raw_import_timestamp AND _raw_batches_._event_day_ = _c_batches_._event_day_ WHERE _c_batches_.raw_import_timestamp IS NULL -- Records in RAW not found in c GROUP BY _raw_batches_._event_day_ ORDER BY _raw_batches_._event_day_ ASC -- Find the oldest record LIMIT 1 -- Return only one import_timestamp/event day batch to ETL
r
Hi! I'm Rocky. It looks like you're running into a syntax or analysis error with your query. While I don't have the specific error message you're seeing, there are a few common issues in this SQL structure that often cause "immediate" errors in StarRocks. 1. Alias in the 
WHERE
Clause In the second subquery (
_c_batches_
), you define an alias 
_event_day_
but use 
event_day
(without the underscores) in the 
WHERE
clause:
Copy code
sql
SELECT 
    _cb_.raw_import_timestamp,
    DATE(CAST(_cb_.`event_time` AS datetime)) AS _event_day_ -- Alias is _event_day_
FROM `events`.`table2` AS _cb_
WHERE event_day >= ... -- Using event_day
Even if you add the underscores, StarRocks (following standard SQL) does not allow using an alias defined in the 
SELECT
list within the same query block's 
WHERE
clause. You must use the original column or the full expression: Fix:
Copy code
sql
WHERE DATE(CAST(_cb_.`event_time` AS datetime)) >= DATE(DATE_SUB(CURDATE(), INTERVAL 15 DAY))
2. Column References and Keywords * Reserved Keywords: Ensure that any column or table names that are reserved keywords are enclosed in backticks. You've already done this for `` 
events
.
table
`` and `` 
event_time
``, which is correct. * Markdown/Formatting: If the 
_
symbols (like 
_SELECT*
) are actually part of your SQL text, they will cause a syntax error. Assuming these are just formatting from your message, you can ignore this. 3. Date Function Precision In Example 8 of the StarRocks 
DATE_TRUNC
documentation, it notes that 
DATE_TRUNC
can return an error if the format doesn't match the data type. In your first subquery: `DATE_TRUNC('day', CAST(
event_time
AS DATETIME))` This is generally safe for the 
'day'
unit, but ensure 
event_time
is castable to 
DATETIME
. 4. 
ARRAY_AGG
Data Types 
ARRAY_AGG
supports 
DATETIME
, 
DATE
, 
VARCHAR
, and other standard types. If 
raw_import_timestamp
is a more complex type (like 
JSON
), it might not be supported depending on your StarRocks version. ────────── Suggested Corrected Query Try replacing the subquery logic to avoid using the alias in the 
WHERE
clause:
Copy code
sql
SELECT 
    _raw_batches_._event_day_, 
    ARRAY_AGG(_raw_batches_.raw_import_timestamp) AS _raw_import_timestamps_
FROM (
    SELECT
        _rb_.raw_import_timestamp,
        DATE(_rb_.`event_time`) AS _event_day_
    FROM `events`.`table` AS _rb_
    WHERE _rb_.raw_import_timestamp < DATE_ADD(NOW(), INTERVAL -2 MINUTE)
      AND DATE_TRUNC('day', CAST(`event_time` AS DATETIME)) >= DATE(DATE_SUB(CURDATE(), INTERVAL 15 DAY))
    GROUP BY _rb_.raw_import_timestamp, DATE(_rb_.`event_time`)
) AS _raw_batches_
LEFT OUTER JOIN (
    SELECT
        _cb_.raw_import_timestamp,
        DATE(CAST(_cb_.`event_time` AS DATETIME)) AS _event_day_
    FROM `events`.`table2` AS _cb_
    WHERE DATE(CAST(_cb_.`event_time` AS DATETIME)) >= DATE(DATE_SUB(CURDATE(), INTERVAL 15 DAY))
    GROUP BY _cb_.raw_import_timestamp, DATE(CAST(_cb_.`event_time` AS DATETIME))
) AS _c_batches_
    ON _raw_batches_.raw_import_timestamp = _c_batches_.raw_import_timestamp
    AND _raw_batches_._event_day_ = _c_batches_._event_day_
WHERE _c_batches_.raw_import_timestamp IS NULL
GROUP BY _raw_batches_._event_day_
ORDER BY _raw_batches_._event_day_ ASC
LIMIT 1;
If this doesn't resolve it, could you please provide the exact error message? This will help me pinpoint if it's a resource limit, a specific function compatibility issue, or a version-specific bug. References • sql-reference/sql-statements/table_bucket_part_index/SELECT.md • faq/operation_maintenance_faq.md • knowledge/usage/best-practice.md • sql-reference/sql-statements/keywords.md
Open in Slack
PreviousNext
Powered by