Hi folks,

GenericRowFileWriter class:

  /**
   * Writes the given row into the files.
   */
  public void write(GenericRow genericRow)
      throws IOException {
    _offsetStream.writeLong(_nextOffset);
    byte[] bytes = _serializer.serialize(genericRow);
    _dataStream.write(bytes);
    _nextOffset += bytes.length;
  }

I use GenericRowFileWriter to write GenericRow to the record.data file. It takes several hours to write more than 20 million data. Why is it so slow to write?

Where is it reading from?

segment file stored on s3

Is it reading row by row from S3? Can you try to copy local first and see if that helps

We first copy to local, then read line by line. Make a partition and write to a local file. The suspicion is that jdk serialization is slower.