https://flink.apache.org/ logo
Join Slack
Powered by
Hi Everyone, For one of SQL Flink Job we set the s...
# troubleshooting
n
Hi Everyone, For one of SQL Flink Job we set the source as filesystem (EFS). The day at which job went live i.e. 25th Jan,2023 the data is being continuously read from the EFS and the required event is sinked to Kinesis successfully. But from 26th no data is being read from the EFS by Flink Job though EFS have new files (confirmed by looking at checkpoints that contains only 25th Jan files). The job state is RUNNING and in the logs we are not seeing and failures or error message. Need the input from the community what can be the probable reason which might have caused the issue.
m
Are the new files newly added, or are they overwriting existing files?
(confirmed by looking at checkpoints that contains only 25th Jan files)
I don't understand what you mean, do you mean that you've inspected the actual checkpoints that were made and that the paths for the 25th of Jan files are there, or do you mean that you've looked at checkpoints in general?
n
Thanks for looking into it. New files keep on adding to EFS over time , as of now we are not deleting any files or overwriting any existing files.
I have looked into the checkpoint today .. it contains the file of 25th Jan only. No files or 26th Jan or beyond but EFS have files of 26th and beyond
s
Did you use partitioning when you defined the source table?
n
yes there is partition on basis of event, year , month , date
s
When the table is partitioned the FileSource will only discover new files in the partitions that existed at the startup, it doesn’t discover new partitions.
I guess you can consider it a bug
The solution is to avoid defining any partitions when you create a table.
n
path i.e. is set as part of job looks like 
'path' = '/flink-data-service/user_event',
each day we will add files on the same e.g. can be 
/flink-data-service/user_event/2023/01/25
/flink-data-service/user_event/2023/01/26'
and so on
s
yep, that's perfectly fine, you don't need to change that
just drop the
Copy code
PARTITIONED BY
clause when you define the file system table in Flink
IMO it doesn't really affect anything when it comes to the file source
n
got it thanks .. in the create temporary table we are using 
PARTITIONED BY
clause that is causing the same !!
m
When the table is partitioned the FileSource will only discover new files in the partitions that existed at the startup, it doesn’t discover new partitions.
Do you know if a ticket already exists for that @sap1ens ?
s
Probably not, I’ll double check and create one today
👍 1
Open in Slack
PreviousNext
Powered by