Hi, I'd like to now if it is intentional that SegmentGenerationAndTaskExecutor don't set the config map of the recordReaderSpec. For plugins requiring additional config (like csv delimiter) it could be an issue no ? I had to create my own inheriting Executor that just extract the config using extracPropsMachingPrefix and push the config into the recordReaderSpec. Is it the good way to do a such thing or did I miss something ?

@Seunghyun ^^

You can configure recordReaderSpec to

SegmentGenerationAndTaskExecutor

using the following configs (this needs to be added as part of task config within table config)

*   recordReader.className - Required, the class name of RecordReader.
 *   recordReader.configClassName - Required, the class name of RecordReaderConfig.
 *   recordReader.prop.<keys> - Optional, the configs used to initialize RecordReaderConfig.

that's my point. It doesn't work because SegmentGenerationAndPushTaskExecutor never set the config field of of the RecordReaderSpec. That's what I understand from the code here SegmentGenerationAndPushTaskExecutor:286

I just checked the code, and You’re right. We need to improve the code to read the properties and set it to the

RecordReaderConfig

. We need to add one following line:

recordReaderSpec.setConfigs(IngestionConfigUtils.getRecordReaderProps(taskConfigs));

Yes, sure. With great pleasure

@Bhioncbr Apache ^^