Good day to everyone. Have anyone had an issue whe...
# troubleshooting
s
Good day to everyone. Have anyone had an issue when a Flink query optimizer just drops WHERE clause from a statement and rewrite results instead of filtering them? It happens with Lookup table from Postgres, but works as expected for versioned table from Kafka. The issue is visible in an image with comparing two bottom screen splits looking at tabs named objective_explode and objective_filter. Up to explode everything was fine: First lookup by primary key, unnesting, and second lookup properly took appropriate value we could see in column c49 in objective_explode. And then I just did WHERE clause and as you can see column c49 row 2 in objective_filter tab hat not been filtered out, instead it got 'psm' value. Compare with column c53 where 'count' value stays and indicates that psm value is not a valid one for that record. Technical details are in a thread. The query to filter data from explode view.
Copy code
create temporary view cxq_internal_service_level_indicator_psm_5sec_over_1hour_objective_filter
as
select
    *
from
    cxq_internal_service_level_indicator_psm_5sec_over_1hour_objective_explode slo
where
      slo.measurement_type = 'psm'
  and slo.measurement_flavor = '5s'
  and slo.metric_interval_unit = 'hour'
  and slo.metric_interval_duration = 1
  and slo.nested_type is null;
While logical plan contains expected LogicalFilter node
LogicalFilter(condition=[AND(=($45, _UTF-16LE'psm'), =($46, _UTF-16LE'5s'), =($51, _UTF-16LE'hour'), =($52, 1), IS NULL($57))])
Physical plan has just Calc node with no where part, but instead it just returns literals as is:
LogicalFilter(condition=[AND(=($45, _UTF-16LE'psm'), =($46, _UTF-16LE'5s'), =($51, _UTF-16LE'hour'), =($52, 1), IS NULL($57))])
(сut for brevity)
Complete execution plan
What I tried to do: • use less condition expression components: No effect, Literals are there instead of actual filtering. • move filtering from after lookup query into a lookup join expression: no effect, lookup join node stays, literals become constants to be returned • try with regular table, not a lookup table: works as expected, LogicalFilter node optimized into Calc(...., where=(...)) e.g.
+- [19]:Calc(select=[bsid, CAST(9001 AS INTEGER) AS sdfid, .....], where=[(sdfid = 9001)]),
still fancy optimizer behaviour to return filter value as a literal, but fine as it works.
I found the tracked and already solved issue: https://issues.apache.org/jira/browse/FLINK-33365 Will update connector. Thanks.