:wave: Hello, team!, Looking to improve AVRO deser...
# troubleshooting
v
πŸ‘‹ Hello, team!, Looking to improve AVRO deserialization performance when creating
DataStream<MyObject>
from AVRO stored in S3. The commands used are
Copy code
Path path = new Path(s3Path);
    AvroInputFormat<MyObject> avroInputFormat = new AvroInputFormat<>(path, MyObject.class);
    return env.createInput(avroInputFormat);
The
s3Path
is a folder containing AVRO data with total size of approximately 1400 GB containing ~150 million records.
MyObject
is a deeply nested object. The performance difference between the above and the following is about 1000x (the following is 1000x faster).
Copy code
Path path = new Path(s3Path);
    TextInputFormat textInputFormat = new TextInputFormat(path);
    return env.createInput(textInputFormat);
In my Flink job, when using AVRO deserialization, I repeatedly get the following message
Copy code
Switching to Random IO seek policy
I'd appreciate any suggestions.
h
If you can access the dashboard, take a look at the flame graph to see the most expensive method calls. I was reading-in Parquet files locally using Flink SQL and parsing adhoc data from a JSON string in one of the columns. It was very slow, and the flame graph showed parsing and swallowed exception were the root cause for the poor performance.