David Bryson
05/21/2024, 10:48 PMjoined_processed_tz_keyed_stream = joined_tz_stream \
.key_by(IdKeySelector("joined_tz_stream"), key_type=Types.STRING()) \
.process(DailyMinMaxProcessor(), joined_tz_processed_info) \
.uid("daily_min_max") \
.name("daily_min_max") \
.forward()
joined_tz_table = table_env.from_data_stream(joined_processed_tz_keyed_stream, joined_tz_processed_schema)
joined_with_metadata = table_env.sql_query(f"""
SELECT lc.*, dpa.device_id, dpa.dev_name, dpa.serial_number, dpa.auth_group
FROM {joined_tz_table} AS lc
JOIN {associations_table_with_tz} AS dpa
ON lc.id = dpa.id
""")
joined_table_result = joined_with_metadata.execute_insert(daily_sink_property_map['flink_table_name'])
As you can see here, i'm matching the incoming data with some metadata in another table. This brings my flink stream to a 100% back pressure scenario due to a single sub-task being 100% busy. How do I distribute the work to other subtasks here?
I should note, my 'forward()' doesn't seem to work here, as the action between operators ends up being hash