Is it possible to filter a stream using a table? For every event that comes in from the data stream I want to query a table and execute some logic such as filter/enrich/etc and pass the event down the pipeline. How would I go about doing this?

In this example, do you mean something like “`publish events only when CREDIT_SCORE>400`” where

CREDIT_SCORE

is recorded in a Flink Table?

Exactly, the credit score is coming from a change log type Kafka topic. What do you recommend using for this case?

One option here could be using Async I/O, as briefly covered here https://genesis-aka.net/information-technology/professional/2021/05/04/enrich-your-d[…]onously-using-amazon-kinesis-data-analytics-for-apache-flink/ Usinig the datastream API, the unbounded stream queries an external data source with the score for each message. In your case, where you are using a CDC style stream to capture the creditscore, a Temporal join may be an option ttps://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#temporal-joins You can then filter off the back of this with the WHERE clause

While I was looking into the subject I found that in this video (99 Ways to Enrich Streaming Data with Apache Flink), he talks about keying the two streams, connecting them then finally using a stateful CoProcessFunction. For my use case, I don't really need a versioned credit score table and I don't care about time accuracy. My events are either published or they aren't and that's not critical to the system. Would you still recommend a Table API Temporal join knowing this? If I were to use the Table API, I would need to convert the event stream to a table. Does that mean Flink is keeping a state of all events it has seen even when I don't need it for my use case?