Next q: I need (in PyFlink) a function that can both emit metrics and also produce to a side output. Is this not possible?

@Andrew Otto Yes, this is supported. • Metrics: see the sample code below:

class MyProcessFucntion(ProcessFucntion):
    def __init__(self):
        self.counter = None

    def open(self, runtime_context: RuntimeContext): 
        self.counter = runtime_context.get_metric_group().counter("my_counter")

    def process_element(self, value, ctx: 'ProcessFunction.Context'):
        self.counter.inc(1)

See more details on https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/python/table/metrics/ (PS: a little tricky here why referring to Table API doc, however, it shares the same usage for metrics between Table API and DataStream API) • Side output: I don’t see why you think it may have problems for side output when working with metrics. Per my understanding, it should just work.

Oh! Great thank you. I will try that. I just assumed that because in Java you needed RichProcessFunction for sideouput + metrics, and I didn’t see RichProcessFunction in python, that it wouldn’t be available.

Yes, submit PR to github/apache-flink

@Dian Fu re async pyflink in Flink 1.18. Do you have a guess on when this might be available? 6 months? 2 years? 🙂