Hi team,
I am trying to read a parquet file on HD...
# troubleshootingz
Zhiyu Tian
05/03/2023, 3:46 AMHi team,
I am trying to read a parquet file on HDFS. But I got the following error
*org.apache.parquet.format.PageHeader: Required field 'uncompressed_page_size' was not found in serialized data*Zhiyu Tian
05/03/2023, 3:51 AM2023-05-03 032618,243 DEBUG org.apache.parquet.hadoop.ParquetFileReader [] - Finished to read all footer bytes.
2023-05-03 032618,243 DEBUG org.apache.parquet.format.converter.ParquetMetadataConverter [] - FileMetaData(version:1, schema:[SchemaElement(name:Microsoft.BI.CFR.Spark.Schema.SiphonLog, num_children:1), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload, field_id:1)], num_rows:49194, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[payload], codec:LZ4, num_values:49194, total_uncompressed_size:164418362, total_compressed_size:62865696, data_page_offset:4, statistics:Statistics(), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:139)]), offset_index_offset:62865700, offset_index_length:1921)], total_byte_size:164418362, num_rows:49194, file_offset:4, total_compressed_size:62865696, ordinal:0)], key_value_metadata:[KeyValue(key:parquet.proto.descriptor, valuename "SiphonLog"
field {
name: "payload"
number: 1
label: LABEL_OPTIONAL
type: TYPE_BYTES
}
), KeyValue(key:parquet.proto.writeSpecsCompliant, value:false), KeyValue(key:writer.model.name, value:protobuf), KeyValue(key:parquet.proto.class, value:Microsoft.BI.CFR.Spark.Schema.SiphonLogOuterClass$SiphonLog)], created_by:parquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b), column_orders:[<ColumnOrder TYPE_ORDER:TypeDefinedOrder()>])
2023-05-03 032618,245 DEBUG org.apache.parquet.format.converter.ParquetMetadataConverter [] - {
"fileMetaData" : {
"schema" : {
"name" : "Microsoft.BI.CFR.Spark.Schema.SiphonLog",
"repetition" : "REPEATED",
"logicalTypeAnnotation" : null,
"id" : null,
"fields" : [ {
"name" : "payload",
"repetition" : "OPTIONAL",
"logicalTypeAnnotation" : null,
"id" : {
"id" : 1
},
"primitive" : true,
"typeLength" : 0,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"originalType" : null
} ],
"columns" : [ {
"path" : [ "payload" ],
"type" : "BINARY",
"primitiveType" : {
"name" : "payload",
"repetition" : "OPTIONAL",
"logicalTypeAnnotation" : null,
"id" : {
"id" : 1
},
"primitive" : true,
"typeLength" : 0,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"originalType" : null
},
"typeLength" : 0,
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0
} ],
"paths" : [ [ "payload" ] ],
"primitive" : false,
"fieldCount" : 1,
"originalType" : null
},
"keyValueMetaData" : {
"parquet.proto.descriptor" : "name: \"SiphonLog\"\nfield {\n name: \"payload\"\n number: 1\n label: LABEL_OPTIONAL\n type: TYPE_BYTES\n}\n",
"parquet.proto.class" : "Microsoft.BI.CFR.Spark.Schema.SiphonLogOuterClass$SiphonLog",
"parquet.proto.writeSpecsCompliant" : "false",
"writer.model.name" : "protobuf"
},
"createdBy" : "parquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)",
"fileDecryptor" : null
},
"blocks" : [ {
"columns" : [ {
"rowGroupOrdinal" : 0,
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"columnIndexReference" : null,
"offsetIndexReference" : {
"offset" : 62865700,
"length" : 1921
},
"bloomFilterOffset" : -1,
"dictionaryPageOffset" : 0,
"valueCount" : 49194,
"totalSize" : 62865696,
"totalUncompressedSize" : 164418362,
"statistics" : {
"max" : null,
"min" : null,
"minBytes" : null,
"maxBytes" : null,
"empty" : true,
"numNulls" : -1,
"numNullsSet" : false
},
"firstDataPageOffset" : 4,
"type" : "BINARY",
"path" : [ "payload" ],
"startingPos" : 4,
"primitiveType" : {
"name" : "payload",
"repetition" : "OPTIONAL",
"logicalTypeAnnotation" : null,
"id" : {
"id" : 1
},
"primitive" : true,
"typeLength" : 0,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"originalType" : null
},
"codec" : "LZ4",
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ]
} ],
"rowCount" : 49194,
"totalByteSize" : 164418362,
"path" : null,
"ordinal" : 0,
"compressedSize" : 62865696,
"startingPos" : 4
} ]
}
2023-05-03 032618,246 INFO org.apache.hadoop.hdfs.DFSClient [] - [DFSInputStream for file /user/zhiyutian/flink-exmples/ads/part-0.parquet]: BP-1520273214-10.27.194.109-1568248216274:blk_52162250811_51120139836 READ 188599096B: [Pure IO Perf 833ms 215.9MB/s] [User E2E Perf 4939ms 36.4MB/s] [PERCENTILE,EXPONENTIAL] [From DN DatanodeInfoWithStorage[/BN2/10/308/09/[260310b0515206a00645be620]10010,DS-07f646f6-dcbf-4b98-8580-1adc1911da0b,DISK]]
2023-05-03 032618,253 DEBUG org.apache.parquet.crypto.DecryptionPropertiesFactory [] - DecryptionPropertiesFactory is not configured - name not found in hadoop config
14 Views