#1886 Add LZO compression support to gpfdist New discussion created by ZTE-EBASE Description This PR adds LZO compressed file format read/write support to the

gpfdist

utility in Apache Cloudberry. Key Insight: LZO is specifically designed for speed over compression ratio. It's ideal for scenarios where decompression throughput matters more than storage savings - such as large-scale data loading into MPP databases like Cloudberry. Core Features Read Support (.lzo → Cloudberry): • ✅ Block-level decompression: Processes LZO data block by block, extracting uncompressed length and compressed length from each block header • ✅ Checksum verification: Validates data integrity using Adler32 and CRC32 checksums embedded in LZO blocks • ✅ Enhanced error reporting: Provides detailed, actionable error messages for various failure scenarios (truncated files, corrupt data, invalid format, checksum mismatches) Write Support (Cloudberry → .lzo): • ✅ Table data unloading: Exports table data to LZO-compressed external files • ✅ Standard LZOP format output: Generates files compatible with standard lzop tool and Hadoop LZO readers • ✅ Configurable compression level: Supports LZO compression levels (default: level 1, fastest; configurable up to level 9) Future Potential - Hadoop Ecosystem Integration: • 🔮 Direct query of Hive tables stored in LZO-compressed format on HDFS • 🔮 Seamless integration with Spark LZO Parquet output • 🔮 Accelerated data loading from Sqoop LZO imports Use case/motivation Scenario 1: High-Volume Data Loading LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio. Scenario 2: Hadoop Ecosystem Integration Many Hadoop ecosystem tools natively support LZO compression: 1. Hive: Supports LZO-compressed text and sequence files 2. Spark: Can write LZO-compressed output (via hadoop-lzo library) 3. Sqoop: Commonly uses LZO for bulk imports from RDBMS to HDFS References Technical Documentation • LZO Official Website • LZOP File Format Specification • Hadoop LZO Compression Guide Cloudberry Documentation • GPFDIST Official Docs • External Tables Guide Related Projects • Apache Cloudberry GitHub • Greenplum Database (upstream) Related issues No response Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry