Hi Team,. I have parquet 28 parquet files in a loc...
# generaln
Nikhil Thomas
11/07/2023, 8:01 AMHi Team,. I have parquet 28 parquet files in a location and while doing the spark segement ingestion , ( num of executor 8, executor core 5) , what I have observed that 19 task is getting lauached
Nikhil Thomas
11/07/2023, 8:03 AMAnd in task is not distributed properlt like 3 of the executor is idle and rest have 3 or 4 task in which
Nikhil Thomas
11/07/2023, 8:03 AMWhat would be the reason
Nikhil Thomas
11/07/2023, 8:05 AMIMG_20231107_133409__01__01.jpg
Nikhil Thomas
11/07/2023, 8:14 AMI have used
Nikhil Thomas
11/07/2023, 8:14 AMto ensure segment created based on my partition column.
x
Xiang Fu
11/07/2023, 8:35 AM
How do you launch the spark job ?
Xiang Fu
11/07/2023, 8:36 AM
If all the files are under same directory, then it should be random distributed. From the log, did you find all the files got processed?
Xiang Fu
11/07/2023, 8:37 AM
Btw, in your case, if there is only one partition, then it’s equivalent to no partition