Good afternoon everyone We have problems with very...
# troubleshooting
a
Good afternoon everyone We have problems with very high latencies when we execute out Flink job (1.12 version) that executes a SQL query with upsert-kafka connector. The latency generated between reading and writing is quite a few hours. The problem arises when the job receives around 800 events per second in each topic. We have a query that cannot be changed with the same structure as this one:
Copy code
SELECT
 …
 
FROM TABLE_A a
   INNER JOIN TABLE_B b
       ON a.atrr1 = b.attr2
 
   INNER JOIN TABLE_C c
       ON c.attr4 = a.attr3
 
   LEFT JOIN TABLE_D d
       ON b.attr5 = d.attr6
 
   LEFT JOIN TABLE_D dd
       ON a.attr7 = dd.attr8
 
   INNER JOIN TABLE_E e
       ON e.attr9 = a.attr10
 
   LEFT JOIN TABLE_F f
       ON f.attr11 = e.attr12
 
   LEFT JOIN TABLE_G g
       ON g.attr13 = e.attr14
 
   INNER JOIN TABLE_H h
       ON h.attr15 = e.attr16
We have tried to set the next parameters in the configuration:
TableConfig tableConfig = tableEnv.getConfig();
Configuration configuration = tableConfig.getConfiguration();
configuration.setString("table.dynamic-table-options.enabled", "true");
configuration.setString("table.exec.mini-batch.enabled", "true");
configuration.setString("table.exec.mini-batch.allow-latency", "2 s");
configuration.setString("table.exec.mini-batch.size", "5000");
configuration.setString("table.exec.async-lookup.buffer-capacity", "10000");
configuration.setString("table.optimizer.distinct-agg.split.enabled", "true");
configuration.setString("table.optimizer.join-reorder-enabled", "true");
tableConfig.addConfiguration(configuration);
To prevent repeated events from collapsing us, we have also created views of the tables which receive more events per second, like this one:
tableEnv.executeSql("CREATE VIEW TABLE_A_VIEW AS SELECT *," + "ROW_NUMBER() OVER (PARTITION BY ROW_ID ORDER BY TransformDateStringToLongFunction(origin_timestamp) DESC) AS row_num " + "FROM TABLE_A");
Even so, the latency remains extremely high. Perhaps it is a normal situation since we are joining so many tables between them, but we want to exhaust all possibilities. Can anyone help us? Thank you very much and greetings.