Looking for some design advice; will appreciate an...
# troubleshooting
s
Looking for some design advice; will appreciate any ideas. (If this is not the right channel for such questions, please let me know.) I need to process a file from an S3 bucket, and write to some database: S3 file ----> My Flink Job ----> SQL db This is not a problem. However, I need to do this periodically, but not on a regular schedule (i.e., cannot use cron, for instance). The source data appears in the S3 bucket every few hours (with no known schedule), and I need to write it to a new table every time. So this is basically a "*stream of batch jobs*" -- some data will appear, needs to be processed as a batch, and then we wait for the next source batch to appear. A "marker" file is written to indicate that data is available in the source bucket (at some location). This will kick off the job, but then how do I go back to "watching mode" where I wait for the next marker file to appear? My understanding is that once I call
env.execute()
, I cannot run any other code after that. How do I structure my Flink job to allow this flow? Thank you for any ideas. PS -- @David Anderson -- been watching your videos, sorry to tag you, but would appreciate any input!
j
Not sure if it's a good idea to run flink 24/7 for processing files once in every several hours. Doesn't S3 fire events when objects are created?
s
Good question...turns out S3 objects are not serializable, so when I attempted to use them from within my job graph, it failed.
j
Not a question but suggestion to look for AWS features beforehand.
πŸ‘ 1
s
Thank you. I will look into that.
c
Is the use of Flink really necessary? Have you considered a Lambda function triggered by the creation of a file in S3? https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html