curious. has anyone ever done a api endpoint "conn...
# randomg
George Leonard
06/25/2025, 5:31 AMcurious. has anyone ever done a api endpoint "connector" in a flink, I mean a endpoint running as one or more jobs on a flink cluster, each on it's own port, to which data is pushed.
d
Danny Cranmer
06/27/2025, 7:43 PMDanny Cranmer
06/27/2025, 7:47 PMThis project is obviously very old. Generally speaking Flink might be overkill for this as a datasource. Since, checkpoints and exactly once guarantees no longer make sense/are possible. You are better off pushing to a proper data store and consuming that in Flink
g
George Leonard
06/28/2025, 7:02 AMFor this data source I don’t want to go via a db. It makes no sense and it’s extra cost.
d
Danny Cranmer
06/28/2025, 7:04 AMOk, but if the Flink job fails you could lose the data/command
g
George Leonard
06/28/2025, 7:10 AMValid. Rethinking this… but more because if we think of Volume velocity variation it might not be good idea to do via api… the idea is to create a spolog end point.
d
Danny Cranmer
06/28/2025, 7:12 AMYeah you would need to push backpressure upstream and get clients to retry. The nice thing (albeit more expensive) about the intermediate data store is that the compute and storage scale independently
g
George Leonard
06/28/2025, 7:15 AMThe easiest actually to use the Beats logger framework. Push onto Kafka and then consume from there.
George Leonard
06/28/2025, 7:15 AMJust figured less tech is less to break and less costs faster end to end time