Hello Flink Friends, I'm trying to understand how ...
# troubleshooting
d
Hello Flink Friends, I'm trying to understand how to troubleshoot a performance problem when going from datastream to table, my table join seems to be concentrating on single subtask for my operator and I don't know why or how to fix it. Many of the docs reference how to optimize processors passing data to other processors, but the actions for table to stream/table operations is less clear. My stream looks something like this:
Copy code
joined_processed_tz_keyed_stream = joined_tz_stream \
            .key_by(IdKeySelector("joined_tz_stream"), key_type=Types.STRING()) \
            .process(DailyMinMaxProcessor(), joined_tz_processed_info) \
            .uid("daily_min_max") \
            .name("daily_min_max") \
            .forward()

    joined_tz_table = table_env.from_data_stream(joined_processed_tz_keyed_stream, joined_tz_processed_schema)

    joined_with_metadata = table_env.sql_query(f"""
                    SELECT lc.*, dpa.device_id, dpa.dev_name, dpa.serial_number, dpa.auth_group
                    FROM {joined_tz_table} AS lc
                    JOIN {associations_table_with_tz} AS dpa
                    ON lc.id = dpa.id
                """)

    joined_table_result = joined_with_metadata.execute_insert(daily_sink_property_map['flink_table_name'])
As you can see here, i'm matching the incoming data with some metadata in another table. This brings my flink stream to a 100% back pressure scenario due to a single sub-task being 100% busy. How do I distribute the work to other subtasks here? I should note, my 'forward()' doesn't seem to work here, as the action between operators ends up being hash