Looking for some design advice; will appreciate any ideas. (If this is not the right channel for such questions, please let me know.)
I need to process a file from an S3 bucket, and write to some database:
S3 file ----> My Flink Job ----> SQL db
This is not a problem. However, I need to do this periodically, but not on a regular schedule (i.e., cannot use cron, for instance).
The source data appears in the S3 bucket every few hours (with no known schedule), and I need to write it to a
new table every time. So this is basically a "*stream of batch jobs*" -- some data will appear, needs to be processed as a batch, and then we wait for the next source batch to appear. A "marker" file is written to indicate that data is available in the source bucket (at some location). This will kick off the job, but then how do I go back to "watching mode" where I wait for the next marker file to appear? My understanding is that once I call
env.execute()
, I cannot run any other code after that.
How do I structure my Flink job to allow this flow?
Thank you for any ideas.
PS --
@David Anderson -- been watching your videos, sorry to tag you, but would appreciate any input!