I'm trying to figure out what's goin on in Flink, ...
# troubleshooting
d
I'm trying to figure out what's goin on in Flink, as my current performance for JDBC lookups is terrible. My goal is use a cached lookup join to enrich a data stream with some metadata. My query is something like the following:
Copy code
SELECT
                             lc.*,
                            get_timezone(dpa.latitude, dpa.longitude) AS timezone
                    FROM incoming_data AS lc
                    JOIN metadata_table
                    FOR SYSTEM_TIME AS OF lc.event_time AS dpa
                    ON lc.id = dpa.id
This query is multiple times slower than every other operator in my stream, and as far as I can tell this is due to non-cached database lookups. I have attempted to configure the source connector for lookup cache:
Copy code
CREATE TABLE metadata_table (
    ID STRING,
    latitutde DOUBLE,
    longitude DOUBLE,
    `timestamp` TIMESTAMP(3),
     WATERMARK FOR `timestamp` AS `timestamp` - INTERVAL '5' SECOND,
     PRIMARY KEY (plant_id) NOT ENFORCED
    ) WITH (
      'connector' = 'jdbc',
      'url' = 'jdbc:postgresql://{2}:5432/{5}',
      'username' = '{3}',
      'password' = '{4}',
      'table-name' = '{6}.{7}',
      'lookup.cache' = 'PARTIAL',
      'lookup.partial-cache.expire-after-write' = '15 min',
      'lookup.partial-cache.max-rows' = '10000'
During my query planner I see this:
TemporalJoin[22] -> Calc[23] -> PythonCalc[24] -> TableToDataSteam
I do not see lookup join, and looking through my logs I don't see anything about caching or lookup. Really confused about why flink is not configuring this as a lookup join. I'm using pyflink 1.18.1.
My issue here appeared to be that LookupJoin will only occur if you are doing lookups based on proctime.