http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I have using StarRocks versions 4.1...
# questions-and-troubleshooting
s
@Rocky I have using StarRocks versions 4.1.4, I am using external glue catalog to manage hive tables. Normal hive tables are working working fine but I am not able to see partitions on glue when I am trying to create a partitioned hive table via StarRocks, I am getting following error:
Copy code
SQL Error [1064] [42000]: Hive metastore did not return partition [merchant_id=120/filter_id=1] for table segmentation.test_part2 (requested 1 partitions, got 0)
When I am trying to run the query:
Copy code
INSERT INTO xeno_hive.segmentation.test_part2
PARTITION(merchant_id=120, filter_id=1)
SELECT 42527300 as customer_id, NOW() as created_at;
r
This error indicates that during the execution of the 
INSERT INTO ... PARTITION
statement, StarRocks successfully identified the target partition but failed to retrieve its metadata from the AWS Glue Catalog (the Hive Metastore equivalent). This typically happens because the partition does not yet exist in Glue, or StarRocks' validation step is failing to "see" it due to Glue's eventual consistency or configuration issues. In StarRocks version 4.1.x, there is stricter validation for static partition clauses in 
INSERT
statements to ensure data integrity. Potential Causes and Solutions 1. Missing Glue Permissions To insert into a partitioned Hive table via Glue, the IAM role/user used by StarRocks must have permissions to both create and retrieve partitions. * Required Permissions: * 
glue:GetTable
* 
glue:GetPartitions
/ 
glue:GetPartition
* 
glue:CreatePartition
(Required if the partition does not exist yet) * 
s3:PutObject
(To write the data files) 2. Glue Partition Projection If your Hive table has Partition Projection enabled in Glue (common for high-cardinality or time-series data), Glue does not actually store partition metadata in its catalog; it calculates them on the fly. * Problem: StarRocks may attempt to verify the partition's existence in the catalog. Since projected partitions aren't "registered" in the Glue metadata store, the metastore returns 0 partitions. * Solution: Check if the table has 
projection.enabled = true
in its Glue properties. If it does, you may need to register partitions manually or ensure StarRocks is configured to handle projected partitions for this catalog (check for properties like 
aws.glue.partition_projection.enabled
in your catalog configuration). 3. Metadata Cache Consistency StarRocks caches external catalog metadata to improve performance. If you just created the table or recently modified partitions, the cache might be out of sync. * Solution: Clear the cache for the table before retrying:
Copy code
sql
    REFRESH EXTERNAL TABLE xeno_hive.segmentation.test_part2;
Alternatively, temporarily disable the external cache for your session:
Copy code
sql
    SET enable_external_catalog_level_cache = false;
4. Case Sensitivity and Data Types Glue partition keys are often case-sensitive and sensitive to formatting. * Check Names: Ensure 
merchant_id
and 
filter_id
in your StarRocks query match the exact casing in the Glue Catalog. * Check Types: If the partition columns in Glue are defined as 
string
but you are passing integers (e.g., 
120
), StarRocks might be requesting the partition using a format Glue doesn't recognize. Try quoting the values in the partition clause:
Copy code
sql
    INSERT INTO xeno_hive.segmentation.test_part2
    PARTITION(merchant_id='120', filter_id='1') ...
5. "Dynamic Overwrite" Setting If you are trying to overwrite data, StarRocks might be looking for existing partitions. You can try enabling dynamic overwrite to allow StarRocks to create missing partitions more flexibly:
Copy code
sql
SET dynamic_overwrite = true;
Recommendation If the issue persists, verify if the partition was actually created in the AWS Glue Console after the failed query. * If the partition IS in Glue: It is a visibility/consistency issue. Try increasing 
aws.glue.metadata_sync_wait_ms
(if available in your specific build) or disabling the FE metadata cache. * If the partition IS NOT in Glue: StarRocks failed to create it. Check your FE logs (
fe.log
) for "Access Denied" or "EntityNotFound" errors from the AWS SDK. References • faq/Sql_faq.md • sql-reference/information_schema/partitions.md • knowledge/admin_and_operation/shared_data_compaction.md
s
@Rocky Here are my Glue table properties:
Copy code
{
  "Name": "test_part2",
  "DatabaseName": "segmentation",
  "Owner": "root",
  "CreateTime": "2026-08-31T06:47:17.000Z",
  "UpdateTime": "2026-08-31T06:47:17.000Z",
  "LastAccessTime": "1970-01-01T00:00:00.000Z",
  "Retention": 0,
  "StorageDescriptor": {
    "Columns": [
      {
        "Name": "customer_id",
        "Type": "bigint"
      },
      {
        "Name": "created_at",
        "Type": "timestamp"
      }
    ],
    "Location": "<s3://segmentation-prod/test_part2>",
    "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
    "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
    "Compressed": false,
    "NumberOfBuckets": 0,
    "SerdeInfo": {
      "Name": "test_part2",
      "SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
    },
    "SortColumns": [],
    "Parameters": {},
    "StoredAsSubDirectories": false
  },
  "PartitionKeys": [
    {
      "Name": "merchant_id",
      "Type": "int"
    },
    {
      "Name": "filter_id",
      "Type": "bigint"
    }
  ],
  "TableType": "MANAGED_TABLE",
  "Parameters": {
    "transient_lastDdlTime": "1788158837",
    "totalSize": "0",
    "numRows": "0",
    "starrocks_version": "4.1.4-4a9848e",
    "starrocks_query_id": "01a05692-833c-79ef-b6fa-f41b34dc7db7"
  },
  "CreatedBy": "arn:aws:sts::<aws_account_id>:assumed-role/starrocks-role/aws-sdk-java-1788123699285",
  "IsRegisteredWithLakeFormation": false,
  "CatalogId": "<aws_account_id>",
  "IsRowFilteringEnabled": false,
  "VersionId": "0",
  "CatalogIdentifier": "<aws_account_id>",
  "DatabaseId": "<glue_db_id>",
  "IsMultiDialectView": false,
  "IsMaterializedView": false
}
I also checked IAM permissions, all the permissions are in place. I see following errors as well in the FE logs:
Copy code
Failed to add partitions on segmentation.test_part2

An exception occurred when using the current long link to access metastore. msg: Failed to add partitions on segmentation.test_part2
r
The error 
SQL Error [1064] [42000]: Hive metastore did not return partition [...] (requested 1 partitions, got 0)
in StarRocks 4.1.x usually occurs because StarRocks attempts to validate that a partition exists in the AWS Glue Catalog immediately after trying to create it during an 
INSERT
operation. If the creation (
add_partition
) fails or if Glue's eventual consistency prevents the partition from being visible immediately, StarRocks throws this error. Based on your logs and table properties, here is the analysis and the steps to resolve it: 1. Change Table Type to 
EXTERNAL_TABLE
 Your Glue table is currently defined as a 
MANAGED_TABLE
. While Glue supports this, most external engines (like StarRocks, Trino, or Spark) expect Hive-compatible tables on S3 to be 
EXTERNAL_TABLE
. Glue's behavior for 
add_partition
on managed tables can sometimes differ, leading to the "Failed to add partitions" error in the FE logs. Solution: Recreate the table as an external table or modify the Glue properties to set 
TableType
to 
EXTERNAL_TABLE
and add the parameter 
"EXTERNAL" = "TRUE"
. 2. Verify IAM Permissions for 
BatchCreatePartition
 The FE log 
Failed to add partitions
indicates that the 
add_partition
RPC call to Glue failed. Even if you have "all permissions," specifically ensure the IAM role used by the StarRocks FE has: * 
glue:BatchCreatePartition
* 
glue:CreatePartition
* 
glue:GetPartition
* 
glue:GetPartitions
Note: In StarRocks, the FE performs the metadata update (adding partitions to Glue), while the BE writes the data to S3. Ensure the FE's IAM role has sufficient Glue permissions. 3. Use Dynamic Partitioning Instead of Static Static partition inserts (
PARTITION(merchant_id=120, ...)
) trigger a specific validation path in StarRocks 4.1.x that is stricter. Using dynamic partitioning (omitting the 
PARTITION
clause and including the partition values in the 
SELECT
list) often bypasses this specific validation check and allows the engine to handle partition creation more flexibly. Try this query instead:
Copy code
sql
INSERT INTO xeno_hive.segmentation.test_part2
-- Note: No PARTITION clause here
SELECT 
    42527300 as customer_id, 
    NOW() as created_at,
    120 as merchant_id, -- Partition columns must be at the end
    1 as filter_id;
4. Check for Location Trailing Slashes If your table location 
<s3://segmentation-prod/test_part2>
does not have a trailing slash, but StarRocks expects one (or vice versa), the generated path for the new partition might be invalid (e.g., `s3://segmentation-prod/test_part2merchant_id=120/...`), causing Glue to reject the 
add_partition
call. * Check the 
location
in your 
CREATE TABLE
statement. * Try adding/removing the trailing slash in the catalog or table configuration. 5. Disable Metadata Caching Temporarily If the partition was actually created in Glue but StarRocks still doesn't "see" it, it might be a cache consistency issue. You can disable the cache for your session to verify:
Copy code
sql
SET enable_external_catalog_level_cache = false;
-- If using a version that supports it:
SET enable_metastore_cache = false;
6. "Long Link" Exception The message 
An exception occurred when using the current long link to access metastore
in your logs is a generic wrapper for a failed RPC call to the metastore. If you are using 
hive.metastore.type = "glue"
, StarRocks is using the AWS SDK. If this error persists, double-check that your 
aws.glue.region
is correctly set in the Catalog configuration, as an incorrect region will cause these RPC calls to fail or time out. Recommended Action: If recreating the table as 
EXTERNAL
and using the dynamic 
INSERT
(Step 3) does not work, please check the FE log specifically for the cause of the 
Failed to add partitions
error (e.g., an 
AccessDeniedException
or 
InvalidInputException
from the AWS SDK) which usually follows that log line. References
3 Views
Open in Slack
PreviousNext
Powered by