Hello I'm teaching myself pyflink table api. I cr...
# troubleshooting
j
Hello I'm teaching myself pyflink table api. I created a script that aggregates revenue by seller_id where tumbling window is setup using event time. The source table is a Kafka topic and it keeps fake sales items as shown below.
Copy code
{
  "seller_id": "LNK",
  "quantity": 5,
  "sale_ts": 1677124093560,
  "name": "Toothpaste",
  "product_price": 4.99
}
The issue is it fails to create the aggregated output into the sink tables when the window is based on the event time (
evttime
) - it just stuck forever. On the other hand, it works when it is based on the process time (
proctime
) and I'm quite confused. Can you please check the attached script and inform me how to make it working?
d
Hey @Jaehyeon Kim it maybe relate to this: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/#source-per-partition-watermarks To summary, there may be no data in some partitions of the Kafka topic which makes the watermark doesn’t advance. If this is the case, you could work around it by one of the following solutions: • configure
table.exec.source.idle-timeout
• set the parallelism to 1
j
Thanks @Dian Fu! Let me try it out.
@Dian Fu I'm able to obtain outcomes by adding the following configuration. Thank you so much!
Copy code
table_env.get_config().set("table.exec.source.idle-timeout", "1000")
🎉 1