Hi Team, I have given batchingestion cofigs as be...
# troubleshooting
p
Hi Team, I have given batchingestion cofigs as below for segmentgenerationandpush minion task, we have multiple files on s3 to ingest, but it is ingesting only one file at scheduled time. "batchIngestionConfig":{ "segmentIngestionType":"REFRESH", "segmentIngestionFrequency":"HOURLY", "consistentDataPush": true, "batchConfigMaps":[{ "input.fs.className":"org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region":"us-east-1", "input.fs.prop.iamrolebasedaccessenabled": true, "input.fs.prop.rolearn":"iamrole", "inputDirURI":"s3bucketuri", "includeFileNamePattern":"glob:*/.parquet", "excludeFileNamePattern":"glob:*/.tmp", "inputFormat":"parquet", "segmentNameGenerator.type": "fixed", "segmentNameGenerator.configs.segment.name": "test" }] } The number of files on s3 will vary for every scheduled run. I would like to know how can refresh the table every time with new files of the data on s3 ?
@Mayank Could you please provide your thoughts on this?
m
Tagging @Manish
m
Hi Prasad, Just wanted to check if you have taken look at this doc for SegmentGenerationAndPushTask. We should be scheduling tasks for a files (one file per task and there's file->segment mapping) - https://docs.pinot.apache.org/basics/concepts/components/cluster/minion#segmentgenerationandpushtask
p
@Manish I checked the doc, I didn't provide tableMaxNumTasks in minion task
so that, it can ingest as many files which matches the includeFileNamePattern
REFRESH functionality is working when I provide segment name as fixed. If I don't provide segment name, data is being ADDED instead of REFRESH
If I provide fixed segment name, only one file get ingested when refresh happening instead of all matching files
m
Oh I see, I think its using the same fixed name for all segments and likely uploading to the same segment? How many segments do you see in the table when you use fixed name.
p
one segment name
I believe all the segments are replacing with the last file if there are multiple files because of fixed segment name
m
Exactly. Do you need to use fix segmentName?
p
I don't have to use fixed segment name. If I don't use REFRESH is not happeing, it is keep on adding the data instead of REFRESH
@Manish Could you please suggest what needs to be done here ?
@Manish could you help me on this?
m
Sorry Prasad, I was occupied with some other stuff. You need a way to refresh existing segments. Can segment name be based on the fileName?
Hey Prasad, If the intention is to refresh the segments based on file modification at the source (i.e if we have file->segment mapping), that is unfortunately not available in SegmentGenerationAndPush minion task. However I do see a way to have file->segment mapping using the SegmentCreationAndTarPush job. Below is an example of file based segment generator config (based on file name). With this, you can get the REFRESH functionality i.e whenever file is modified, we will refresh the segment because we use the same segment name. segmentNameGeneratorSpec: type: inputFile configs: file.path.pattern: '.+/dt=(.+)/part-(\d+)-.+\.parquet' segment.name.template: '${table_name}_v${table_version}_\${filePathPattern:\1}-\${filePathPattern:\2}'
p
@Manish I will try and let you know my findings on this in my use case, file name will vary every time, but I will try with both options i) file name with vary ii) same file name
m
1. If you want to just ingest files incrementally, then if you remove the fixed name generator, you'll be able to create segments as you add files. 2. If you want refresh functionality i.e to refresh the same segment and not create a new one, then you'll have to take the file pattern based segment naming approach. However you can achieve 1) with this approach as well.
p
for example, today, if I get added 2 files added to source, the minion task ingest two files data and tomorrow if 3 files added to source, will minion task refresh the data with 3 files or just 2 files of data?