http://starrocks.io logo
Join Slack
Powered by
Hi team, facing one issue with delta table in star...
# questions-and-troubleshooting
g
Hi team, facing one issue with delta table in starrocks , in iceberg it is working fine and giving output, but not in delta table, table as data of 
select '2025-10-22' as d
Copy code
mysql> explain select * from external_delta_table where d = current_date();
+-------------------------------------------------------------------------------+
| Explain String                                                                |
+-------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                               |
|  OUTPUT EXPRS:1: d                                                            |
|   PARTITION: UNPARTITIONED                                                    |
|                                                                               |
|   RESULT SINK                                                                 |
|                                                                               |
|   1:EXCHANGE                                                                  |
|                                                                               |
| PLAN FRAGMENT 1                                                               |
|  OUTPUT EXPRS:                                                                |
|   PARTITION: RANDOM                                                           |
|                                                                               |
|   STREAM DATA SINK                                                            |
|     EXCHANGE ID: 01                                                           |
|     UNPARTITIONED                                                             |
|                                                                               |
|   0:DeltaLakeScanNode                                                         |
|      TABLE: external_delta_table                                              |
|      NON-PARTITION PREDICATES: CAST(1: d AS DATETIME) = '2025-10-22 00:00:00' |
|      cardinality=500000000                                                    |
|      avgRowSize=1.0                                                           |
+-------------------------------------------------------------------------------+
21 rows in set (1.403 sec)


mysql> explain select * from external_iceberg_table where d = current_date();
+-------------------------------------------------------------------------------+
| Explain String                                                                |
+-------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                               |
|  OUTPUT EXPRS:1: d                                                            |
|   PARTITION: UNPARTITIONED                                                    |
|                                                                               |
|   RESULT SINK                                                                 |
|                                                                               |
|   1:EXCHANGE                                                                  |
|                                                                               |
| PLAN FRAGMENT 1                                                               |
|  OUTPUT EXPRS:                                                                |
|   PARTITION: RANDOM                                                           |
|                                                                               |
|   STREAM DATA SINK                                                            |
|     EXCHANGE ID: 01                                                           |
|     UNPARTITIONED                                                             |
|                                                                               |
|   0:IcebergScanNode                                                           |
|      TABLE: external_iceberg_table                                            |
|      PREDICATES: CAST(1: d AS DATETIME) = '2025-10-22 00:00:00'               |
|      cardinality=1                                                            |
|      avgRowSize=10.0                                                          |
+-------------------------------------------------------------------------------+
21 rows in set (1.203 sec)
is there some bug in delta lake be cpp code, can someone help
@Ronit Kapoor can you please help here
r
Which query is not working? Can you also share what the output is?
g
output is empty
empty in delta, 
2025-10-22
in iceberg
anything here @Robert Raharjo
r
Are you running this command and doesn't get the output? 
select '2025-10-22' as d;
This doesn't seem to be related with datalake. Can you share the profile for the failing query?
m
@Robert Raharjo. I create a table in delta_lake using below command 
create table gold.scratch.starrocks_date_test as (select cast(date(current_date())  as string)  as d)
. When i'm querying this table using 
select * from  gold.scratch.starrocks_date_test where d = date(current_date())
query. This query is giving result in iceberg and starrocks(provided table is created in starrocks using same query) native databases but not using delta external catalog.
basically my column datatype is string, and when I'm comparing it with date type. It is working in starrocks and iceberg but not in delta lake connector. eventhough plan of all is same as shared above by @GAURAV MIGLANI
@Robert Raharjo Did you get a chance to look into this.
@Robert Raharjo We need your help here. It has become a blocker for us to us starrocks in production ENV.
r
Yes, can you please share the output and profile for the delta lake table? Have you confirmed that the Delta lake table have the tuple with 
d = current_date()
? We can't help much without knowing what's going on behind the scene.
g
can you explain in detail what exactly all details you need
will provide end to end details, can you share commands please
r
The queries and their result. On top of that can you also share the profile 
set enable_profile = true;
Run the queries, open your browser and go to 
http://<fe_ip>:<fe_http_port>
.Under the queries tab, click on the profiles. This will give you the profiles. Default 
fe_http_port
is 8030 https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview/#via-sql-function-get_query_profile
g
Copy code
Query:
  Summary:
     - Query ID: a44b9185-b75e-11f0-a334-16cc49abb971
     - Start Time: 2025-11-01 20:09:15
     - End Time: 2025-11-01 20:13:15
     - Total: 3m59s
     - Query Type: Query
     - Query State: Finished
     - StarRocks Version: 3.5.5-fd4e51b
     - User: root
     - Default Db: analytics
     - Sql Statement: select * from silver.events.added_to_cart where datestr=current_date() limit 11
     - Variables: parallel_fragment_exec_instance_num=1,max_parallel_scan_instance_num=-1,pipeline_dop=0,enable_adaptive_sink_dop=true,enable_runtime_adaptive_dop=false,runtime_profile_report_interval=10,resource_group=default_wg
     - NonDefaultSessionVariables: {"enable_lake_tablet_internal_parallel":{"defaultValue":true,"actualValue":false},"default_table_compression":{"defaultValue":"lz4_frame","actualValue":"zstd"},"query_mem_limit":{"defaultValue":0,"actualValue":42949672960},"query_timeout":{"defaultValue":300,"actualValue":1200},"dynamic_overwrite":{"defaultValue":false,"actualValue":true},"new_planner_optimize_timeout":{"defaultValue":3000,"actualValue":15000},"enable_spill_to_remote_storage":{"defaultValue":false,"actualValue":true},"enable_adaptive_sink_dop":{"defaultValue":false,"actualValue":true},"enable_spill":{"defaultValue":false,"actualValue":true},"spill_storage_volume":{"defaultValue":"","actualValue":"starrocks_s3_spill_cn"},"enable_read_iceberg_equality_delete_with_partition_evolution":{"defaultValue":false,"actualValue":true},"enable_rewrite_simple_agg_to_hdfs_scan":{"defaultValue":true,"actualValue":false},"enable_drop_table_check_mv_dependency":{"defaultValue":false,"actualValue":true},"enable_multicolumn_global_runtime_filter_v2":{"defaultValue":true,"actualValue":false}}
     - Collect Profile Time: 0
     - IsProfileAsync: true
  Planner:
     - -- Parser[1] 1ms
     - -- Total[1] 3ms
     -     -- Analyzer[1] 0
     -         -- Lock[1] 0
     -         -- AnalyzeDatabase[1] 0
     -         -- AnalyzeTemporaryTable[1] 0
     -         -- AnalyzeTable[1] 0
     -     -- Transformer[1] 0
     -     -- Optimizer[1] 1ms
     -         -- MVPreprocess[1] 0
     -         -- MVTextRewrite[1] 0
     -         -- RuleBaseOptimize[1] 0
     -         -- CostBaseOptimize[1] 0
     -         -- PhysicalRewrite[1] 0
     -         -- DynamicRewrite[1] 0
     -         -- PlanValidate[1] 0
     -             -- InputDependenciesChecker[1] 0
     -             -- TypeChecker[1] 0
     -             -- CTEUniqueChecker[1] 0
     -             -- ColumnReuseChecker[1] 0
     -     -- ExecPlanBuild[1] 0
     - -- Pending[1] 3m59s
     - -- Prepare[1] 674ms
     -     -- DeltaLake.getScanFiles[2] 674ms
     -         -- DeltaLake.getScanFiles[1] 674ms
     -             -- DeltaLakeJsonHandler.JsonToColumnarBatch[18] 0
     -             -- DeltaLakeParquetHandler.readParquetFileAndGetColumnarBatch[2] 519ms
     - -- Deploy[1] 55ms
     -     -- DeployLockInternalTime[1] 55ms
     -         -- DeploySerializeConcurrencyTime[2] 0
     -         -- DeployStageByStageTime[6] 0
     -         -- DeployWaitTime[6] 54ms
     -             -- DeployAsyncSendTime[7] 0
     - DeployDataSize: 180067
    Reason:
  Execution:
     - Topology: {"rootId":1,"nodes":[{"id":1,"name":"EXCHANGE","properties":{"sinkIds":[],"displayMem":true},"children":[0]},{"id":0,"name":"DELTA_LAKE_SCAN","properties":{"sinkIds":[1],"displayMem":false},"children":[]}]}
     - FrontendProfileMergeTime: 50.993us
     - QueryAllocatedMemoryUsage: 0.000 B
     - QueryCumulativeCpuTime: 0ns
     - QueryCumulativeNetworkTime: 0ns
     - QueryCumulativeOperatorTime: 0ns
     - QueryCumulativeScanTime: 0ns
     - QueryDeallocatedMemoryUsage: 0.000 B
     - QueryExecutionWallTime: 0ns
     - QueryPeakMemoryUsagePerNode: 0.000 B
     - QueryPeakScheduleTime: 0ns
     - QuerySpillBytes: 0.000 B
     - QuerySumMemoryUsage: 0.000 B
    Fragment 0:
       - BackendAddresses: data-engineering-cn-cn-1.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060
       - InstanceIds: a44b9185-b75e-11f0-a334-16cc49abb972
       - MissingInstanceIds: a44b9185-b75e-11f0-a334-16cc49abb972
       - BackendNum: 1
       - InstanceNum: 1
    Fragment 1:
       - BackendAddresses: data-engineering-cn-cn-5.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-3.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-0.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-4.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-1.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-2.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060
       - InstanceIds: a44b9185-b75e-11f0-a334-16cc49abb975,a44b9185-b75e-11f0-a334-16cc49abb976,a44b9185-b75e-11f0-a334-16cc49abb977,a44b9185-b75e-11f0-a334-16cc49abb978,a44b9185-b75e-11f0-a334-16cc49abb973,a44b9185-b75e-11f0-a334-16cc49abb974
       - MissingInstanceIds: a44b9185-b75e-11f0-a334-16cc49abb975,a44b9185-b75e-11f0-a334-16cc49abb976,a44b9185-b75e-11f0-a334-16cc49abb977,a44b9185-b75e-11f0-a334-16cc49abb978,a44b9185-b75e-11f0-a334-16cc49abb973,a44b9185-b75e-11f0-a334-16cc49abb974
       - BackendNum: 6
       - InstanceNum: 6
Copy code
Query:
  Summary:
     - Query ID: ab0a8440-b75d-11f0-8db0-3e10fa65ce59
     - Start Time: 2025-11-01 20:02:17
     - End Time: 2025-11-01 20:06:49
     - Total: 4m31s
     - Query Type: Query
     - Query State: Finished
     - StarRocks Version: 3.5.5-fd4e51b
     - User: root
     - Default Db: analytics
     - Sql Statement: select * from silver.events.added_to_cart where datestr='2025-11-01' limit 11
     - Variables: parallel_fragment_exec_instance_num=1,max_parallel_scan_instance_num=-1,pipeline_dop=0,enable_adaptive_sink_dop=true,enable_runtime_adaptive_dop=false,runtime_profile_report_interval=10,resource_group=default_wg
     - NonDefaultSessionVariables: {"enable_lake_tablet_internal_parallel":{"defaultValue":true,"actualValue":false},"default_table_compression":{"defaultValue":"lz4_frame","actualValue":"zstd"},"query_mem_limit":{"defaultValue":0,"actualValue":42949672960},"query_timeout":{"defaultValue":300,"actualValue":1200},"dynamic_overwrite":{"defaultValue":false,"actualValue":true},"new_planner_optimize_timeout":{"defaultValue":3000,"actualValue":15000},"enable_spill_to_remote_storage":{"defaultValue":false,"actualValue":true},"enable_adaptive_sink_dop":{"defaultValue":false,"actualValue":true},"enable_spill":{"defaultValue":false,"actualValue":true},"spill_storage_volume":{"defaultValue":"","actualValue":"starrocks_s3_spill_cn"},"enable_read_iceberg_equality_delete_with_partition_evolution":{"defaultValue":false,"actualValue":true},"enable_rewrite_simple_agg_to_hdfs_scan":{"defaultValue":true,"actualValue":false},"enable_drop_table_check_mv_dependency":{"defaultValue":false,"actualValue":true},"enable_multicolumn_global_runtime_filter_v2":{"defaultValue":true,"actualValue":false}}
     - Collect Profile Time: 32ms
     - IsProfileAsync: true
  Planner:
     - -- Parser[1] 0
     - -- Total[1] 2ms
     -     -- Analyzer[1] 0
     -         -- Lock[1] 0
     -         -- AnalyzeDatabase[1] 0
     -         -- AnalyzeTemporaryTable[1] 0
     -         -- AnalyzeTable[1] 0
     -     -- Transformer[1] 0
     -     -- Optimizer[1] 1ms
     -         -- MVPreprocess[1] 0
     -         -- MVTextRewrite[1] 0
     -         -- RuleBaseOptimize[1] 0
     -         -- CostBaseOptimize[1] 0
     -         -- PhysicalRewrite[1] 0
     -         -- DynamicRewrite[1] 0
     -         -- PlanValidate[1] 0
     -             -- InputDependenciesChecker[1] 0
     -             -- TypeChecker[1] 0
     -             -- CTEUniqueChecker[1] 0
     -             -- ColumnReuseChecker[1] 0
     -     -- ExecPlanBuild[1] 0
     - -- Pending[1] 4m21s
     - -- Prepare[1] 611ms
     -     -- DeltaLake.getScanFiles[2] 610ms
     -         -- DeltaLake.getScanFiles[113] 607ms
     -             -- DeltaLakeJsonHandler.JsonToColumnarBatch[18] 0
     -             -- DeltaLakeParquetHandler.readParquetFileAndGetColumnarBatch[2] 481ms
     - -- Deploy[1] 320ms
     -     -- DeployLockInternalTime[1] 320ms
     -         -- DeploySerializeConcurrencyTime[2] 1ms
     -         -- DeployStageByStageTime[6] 0
     -         -- DeployWaitTime[6] 318ms
     -             -- DeployAsyncSendTime[7] 0
     - DeployDataSize: 198929
    Placement:
      added_to_cart:
        balance:
           - data-engineering-cn-cn-5_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 0
           - data-engineering-cn-cn-3_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 0
           - data-engineering-cn-cn-1_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 0
           - data-engineering-cn-cn-4_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 0
           - data-engineering-cn-cn-0_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 0
           - data-engineering-cn-cn-2_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 0
        assign:
           - data-engineering-cn-cn-3_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 1398137506
           - data-engineering-cn-cn-0_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 1961038070
           - data-engineering-cn-cn-5_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 2218178888
           - data-engineering-cn-cn-2_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 1946604531
           - data-engineering-cn-cn-1_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 2517771241
           - data-engineering-cn-cn-4_data-engineering-cn-cn-search_data-engineering_svc_cluster_local: 2974571189
    Reason:
  Execution:
     - Topology: {"rootId":1,"nodes":[{"id":1,"name":"EXCHANGE","properties":{"sinkIds":[],"displayMem":true},"children":[0]},{"id":0,"name":"DELTA_LAKE_SCAN","properties":{"sinkIds":[1],"displayMem":false},"children":[]}]}
     - FrontendProfileMergeTime: 53.292us
     - QueryAllocatedMemoryUsage: 0.000 B
     - QueryCumulativeCpuTime: 0ns
     - QueryCumulativeNetworkTime: 0ns
     - QueryCumulativeOperatorTime: 0ns
     - QueryCumulativeScanTime: 0ns
     - QueryDeallocatedMemoryUsage: 0.000 B
     - QueryExecutionWallTime: 0ns
     - QueryPeakMemoryUsagePerNode: 0.000 B
     - QueryPeakScheduleTime: 0ns
     - QuerySpillBytes: 0.000 B
     - QuerySumMemoryUsage: 0.000 B
    Fragment 0:
       - BackendAddresses: data-engineering-cn-cn-4.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060
       - InstanceIds: ab0a8440-b75d-11f0-8db0-3e10fa65ce5a
       - MissingInstanceIds: ab0a8440-b75d-11f0-8db0-3e10fa65ce5a
       - BackendNum: 1
       - InstanceNum: 1
    Fragment 1:
       - BackendAddresses: data-engineering-cn-cn-5.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-3.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-0.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-4.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-1.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060,data-engineering-cn-cn-2.data-engineering-cn-cn-search.data-engineering.svc.cluster.local:9060
       - InstanceIds: ab0a8440-b75d-11f0-8db0-3e10fa65ce5f,ab0a8440-b75d-11f0-8db0-3e10fa65ce5e,ab0a8440-b75d-11f0-8db0-3e10fa65ce5d,ab0a8440-b75d-11f0-8db0-3e10fa65ce5c,ab0a8440-b75d-11f0-8db0-3e10fa65ce5b,ab0a8440-b75d-11f0-8db0-3e10fa65ce60
       - MissingInstanceIds: ab0a8440-b75d-11f0-8db0-3e10fa65ce5f,ab0a8440-b75d-11f0-8db0-3e10fa65ce5e,ab0a8440-b75d-11f0-8db0-3e10fa65ce5d,ab0a8440-b75d-11f0-8db0-3e10fa65ce5c,ab0a8440-b75d-11f0-8db0-3e10fa65ce5b,ab0a8440-b75d-11f0-8db0-3e10fa65ce60
       - BackendNum: 6
       - InstanceNum: 6
current_date() when i ran above both queries was '2025-11-01'
@Robert Raharjo can you please help here, what we are missing here
r
Seems like there's a bug with the code. Delta Lake is not supported as much as Iceberg, and currently there's no concrete plan to further develop the Delta Lake compatibility - my suggestion is to switch to Iceberg.
g
why, is starrocks deprecating delta lake support?
2 Views
Open in Slack
PreviousNext
Powered by