David Bryson
05/29/2024, 2:34 AMSELECT
lc.*,
get_timezone(dpa.latitude, dpa.longitude) AS timezone
FROM incoming_data AS lc
JOIN metadata_table
FOR SYSTEM_TIME AS OF lc.event_time AS dpa
ON lc.id = dpa.id
This query is multiple times slower than every other operator in my stream, and as far as I can tell this is due to non-cached database lookups. I have attempted to configure the source connector for lookup cache:
CREATE TABLE metadata_table (
ID STRING,
latitutde DOUBLE,
longitude DOUBLE,
`timestamp` TIMESTAMP(3),
WATERMARK FOR `timestamp` AS `timestamp` - INTERVAL '5' SECOND,
PRIMARY KEY (plant_id) NOT ENFORCED
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:postgresql://{2}:5432/{5}',
'username' = '{3}',
'password' = '{4}',
'table-name' = '{6}.{7}',
'lookup.cache' = 'PARTIAL',
'lookup.partial-cache.expire-after-write' = '15 min',
'lookup.partial-cache.max-rows' = '10000'
During my query planner I see this:
TemporalJoin[22] -> Calc[23] -> PythonCalc[24] -> TableToDataSteam
I do not see lookup join, and looking through my logs I don't see anything about caching or lookup. Really confused about why flink is not configuring this as a lookup join. I'm using pyflink 1.18.1.David Bryson
05/30/2024, 5:46 PM