Hi Folks. I have a question. I am using flink and ...
# troubleshooting
k
Hi Folks. I have a question. I am using flink and scala for my streaming application, it has data coming for all of our VMs, like CPU utilization, memory usage, etc. Now we have 2 category of VMs test and non test VMs. When I read data from Kafka, I need to filter out the data for only non test VMs so that we can monitor those and set up alerting systems in grafana. Now a list of non test VMs is present in DB table and its dynamic. I am using JDBCCatalog with cache to keep a copy of the VMs info handy when streaming data so that I can filter out required data. Now the error I am facing here is, my application is running fine on a platform when consumer parallelism = 8 and processing parallelism = 2(however the application is very slow here), As soon as I increase my processing parallelism to 4 my application starts to fail with the following errors: 1. org.apache.flink.table.api.TableException: Failed to execute sql 2. org.apache.flink.table.api.TableException: Cannot generate a valid execution plan for the given query: This exception indicates that the query uses an unsupported SQL feature.Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=LOGICAL, FlinkRelDistributionTraitDef=any, MiniBatchIntervalTraitDef=None: 0, ModifyKindSetTraitDef=[NONE], UpdateKindTraitDef=[NONE]. Missing conversion is LogicalSink[convention: NONE -> LOGICAL] 3. java.util.ConcurrentModificationException None of these errors are there when processingParallelism = 2 Are these known issues with Flink, have any of you faced this before.