Hey everyone,
My project requires writing a csv file to an S3 bucket which contains headers using the sink functionality of apache flink (version 1.18.0). Programming language used is java. Hadoop file system is being used via flink's library named "flink-s3-fs-hadoop".
Example of CSV data is:
student_id,exam_id,subject,score,grade
1,1,Math,41,D
1,1,Spanish,51,C
The below code is working but it is not writing the headers to the csv file in S3. Code snippet used:
private static final StreamExecutionEnvironment ENV;
private static final StreamTableEnvironment TABLE_ENV;
static {
ENV = StreamExecutionEnvironment.getExecutionEnvironment()
.setRuntimeMode(RuntimeExecutionMode.BATCH)
.setParallelism(1);
TABLE_ENV = StreamTableEnvironment.create(ENV);
}
DataType dataType = DataTypes.ROW(
DataTypes.FIELD("student_id",
DataTypes.INT()),
DataTypes.FIELD("exam_id",
DataTypes.INT()),
DataTypes.FIELD("subject", DataTypes.STRING()),
DataTypes.FIELD("score",
DataTypes.INT()),
DataTypes.FIELD("grade", DataTypes.STRING())
);
RowType rowType = (RowType) dataType.getLogicalType();
CsvRowDataSerializationSchema serSchemaBuilder =
new CsvRowDataSerializationSchema.Builder(rowType).build();
FileSink<RowData> sink = FileSink.forRowFormat(new Path(s3FilePath), new SerializationSchemaAdapter(serSchemaBuilder))
.withOutputFileConfig(new OutputFileConfig("test", ".csv"))
.build();
rowData.sinkTo(sink);
ENV.execute();
Expected Output of the CSV File in S3:
student_id,exam_id,subject,score,grade
1,1,Math,41,D
1,1,Spanish,51,C
Actual Output of the CSV File in S3:
1,1,Math,41,D
1,1,Spanish,51,C
Can anybody please let me know if there is a way to write csv file with headers using the sink functionality?