Vinay Agarwal
01/11/2023, 5:12 PMDataStream<MyObject> from AVRO stored in S3. The commands used are
Path path = new Path(s3Path);
AvroInputFormat<MyObject> avroInputFormat = new AvroInputFormat<>(path, MyObject.class);
return env.createInput(avroInputFormat);
The s3Path is a folder containing AVRO data with total size of approximately 1400 GB containing ~150 million records. MyObject is a deeply nested object. The performance difference between the above and the following is about 1000x (the following is 1000x faster).
Path path = new Path(s3Path);
TextInputFormat textInputFormat = new TextInputFormat(path);
return env.createInput(textInputFormat);
In my Flink job, when using AVRO deserialization, I repeatedly get the following message
Switching to Random IO seek policy
I'd appreciate any suggestions.Hunter Medney
01/14/2023, 8:57 PM