Adan Fernández
06/12/2024, 11:58 AMSELECT
…
FROM TABLE_A a
INNER JOIN TABLE_B b
ON a.atrr1 = b.attr2
INNER JOIN TABLE_C c
ON c.attr4 = a.attr3
LEFT JOIN TABLE_D d
ON b.attr5 = d.attr6
LEFT JOIN TABLE_D dd
ON a.attr7 = dd.attr8
INNER JOIN TABLE_E e
ON e.attr9 = a.attr10
LEFT JOIN TABLE_F f
ON f.attr11 = e.attr12
LEFT JOIN TABLE_G g
ON g.attr13 = e.attr14
INNER JOIN TABLE_H h
ON h.attr15 = e.attr16
We have tried to set the next parameters in the configuration:
TableConfig tableConfig = tableEnv.getConfig();
Configuration configuration = tableConfig.getConfiguration();
configuration.setString("table.dynamic-table-options.enabled", "true");
configuration.setString("table.exec.mini-batch.enabled", "true");
configuration.setString("table.exec.mini-batch.allow-latency", "2 s");
configuration.setString("table.exec.mini-batch.size", "5000");
configuration.setString("table.exec.async-lookup.buffer-capacity", "10000");
configuration.setString("table.optimizer.distinct-agg.split.enabled", "true");
configuration.setString("table.optimizer.join-reorder-enabled", "true");
tableConfig.addConfiguration(configuration);
To prevent repeated events from collapsing us, we have also created views of the tables which receive more events per second, like this one:
tableEnv.executeSql("CREATE VIEW TABLE_A_VIEW AS SELECT *," + "ROW_NUMBER() OVER (PARTITION BY ROW_ID ORDER BY TransformDateStringToLongFunction(origin_timestamp) DESC) AS row_num " + "FROM TABLE_A");
Even so, the latency remains extremely high. Perhaps it is a normal situation since we are joining so many tables between them, but we want to exhaust all possibilities. Can anyone help us?
Thank you very much and greetings.