Anis Shaikh
06/24/2024, 10:38 AMStreamExecutionEnvironment and StreamTableEnvironment.
◦ Enabling object reuse and configuring mini-batch settings.
2. Data Fetching and Schema Creation:
◦ Reading JSON arguments from Flink UI.
◦ Fetching schema and GroupModel data from MongoDB.
◦ Creating a schema for a temporary view.
3. DataStream Transformations:
◦ Creating a DataStream from the CSV file.
◦ Transforming the DataStream into a DataStream of Tuple3String, Row, String.
◦ Further transforming this into a DataStream of Row.
4. Temporary View and Query Execution:
◦ Creating a temporary view from the transformed DataStream.
◦ Iterating over the GroupModel list, modifying and executing SQL queries on the temporary view.
◦ Executing queries on temporary view and getting result as Table.
◦ Combining all Table into a single Table and converting it back into a DataStream of Row.
5. Windowing and Event Processing:
◦ Applying window operations and transformations.
◦ Generating GroupChangeEvent and pushing it to Kafka.
The Issue
The job appears to execute without errors, as seen in the Flink UI. However, no data is being pushed to the Kafka topic. Upon closer inspection, it seems that after the transformation steps, particularly the anonymous DataStream source step (*anonymous_datastream_source$5*[204] -> (Calc[205], Calc[206], Calc[208], ... Calc[302])), data is not being forwarded to subsequent steps.
Here are some specifics:
• DataStream and Transformation Steps: Data appears to be processed up to a certain point, but does not propagate further.
• Logging and Debugging: I have added logging to various transformation steps but have not been able to pinpoint the exact cause of the issue.
What I've Checked
• Kafka Connectivity: Verified that Kafka is accessible and the topic exists.
• Serialization: Confirmed that the data serialization to Kafka is correctly configured.
• Job Execution: The job runs without any visible errors in the logs or the Flink UI.
Has anyone encountered a similar issue where data does not propagate through the DataStream transformations? Are there any best practices or additional debugging steps you could suggest to diagnose and resolve this issue?
Any help or pointers would be greatly appreciated!
Attaching screenshot from flink UI for better visualization.
Thank you in advance for your assistance.Anis Shaikh
06/24/2024, 10:40 AMSaurabh Singh
07/03/2024, 8:15 AM