Hey guys, is there an example or a clear explanati...
# troubleshootingm
Max Dubinin
04/07/2023, 10:59 AMHey guys, is there an example or a clear explanation anywhere of streaming POJOs to Parquet without Avro?
d
David Anderson
04/07/2023, 6:32 PMhttps://docs.immerok.cloud/docs/how-to-guides/development/read-from-apache-kafka-write-to-parquet-files-with-apache-flink/ reads JSON from Kafka, and writes Parquet files.
m
Max Dubinin
04/07/2023, 7:58 PMthank you very much!
Max Dubinin
04/07/2023, 8:00 PMI managed to do so with Table API. My problem was that I didn’t enable checkpointing (I was lucky enough to stumble upon one of your answers in SO to figure that out). Do you know why the Parquet files are being written at the checkpoints, instead of by batching them to 128MB or by the rolling interval as configured in the sink?
d
David Anderson
04/07/2023, 8:58 PMWith bulk (column-based) formats, it would be awkward to provide exactly-once guarantees without synchronizing the bulk writes with checkpointing.
m
Max Dubinin
04/08/2023, 10:43 AMNOTE: For bulk formats (parquet, orc, avro), the rolling policy in combination with the checkpoint interval(pending files become finished on the next checkpoint) control the size and number of these parts.just saw this. Thanks
2 Views