Hi Is row groups supported in parquet file format ingestion

Question

Xiang Fu · Answer

i will take a look into it

Xiang Fu · Answer

I ve created a parquet with row groups and it works

Xiang Fu · Answer

``` parquet tools meta tmp data parquet | grep row group | head n 3 row group 1 RC 93898 TS 131168051 OFFSET 4 row group 2 RC 93921 TS 131157859 OFFSET 131168055 row group 3 RC 93899 TS 131162316 OFFSET 262325914```

Xiang Fu · Answer

``` parquet tools meta tmp data parquet file file tmp data parquet creator parquet mr version 1 8 0 build 0fda28af84b9746396014ad6a415b90592a98b3b extra parquet avro schema = type record name record fields name met float type float name met double type double name dim mv long type type array items long name dim sv double type double name met long type long name dim sv string type string name dim mv int type type array items int name dim mv string type type array items string name dim sv int type int name dim mv double type type array items double name dim mv float type type array items float name dim sv float type float name met int type int name dim sv long type long file schema record met float REQUIRED FLOAT R 0 D 0 met double REQUIRED DOUBLE R 0 D 0 dim mv long REQUIRED F 1 array REPEATED INT64 R 1 D 1 dim sv double REQUIRED DOUBLE R 0 D 0 met long REQUIRED INT64 R 0 D 0 dim sv string REQUIRED BINARY O UTF8 R 0 D 0 dim mv int REQUIRED F 1 array REPEATED INT32 R 1 D 1 dim mv string REQUIRED F 1 array REPEATED BINARY O UTF8 R 1 D 1 dim sv int REQUIRED INT32 R 0 D 0 dim mv double REQUIRED F 1 array REPEATED DOUBLE R 1 D 1 dim mv float REQUIRED F 1 array REPEATED FLOAT R 1 D 1 dim sv float REQUIRED FLOAT R 0 D 0 met int REQUIRED INT32 R 0 D 0 dim sv long REQUIRED INT64 R 0 D 0 row group 1 RC 93898 TS 131168051 OFFSET 4 met float FLOAT UNCOMPRESSED DO 0 FPO 4 SZ 204570 204570 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 1 2111664E 4 max 0 9999642 num nulls 0 met double DOUBLE UNCOMPRESSED DO 0 FPO 204574 SZ 244586 244586 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 1 7748871805833843E 5 max 0 9999900631857935 num nulls 0 dim mv long array INT64 UNCOMPRESSED DO 0 FPO 449160 SZ 19375790 19375790 1 00 VC 2386112 ENC RLE PLAIN ST min 9223315962872288953 max 9223235375325574819 num nulls 0 dim sv double DOUBLE UNCOMPRESSED DO 0 FPO 19824950 SZ 244586 244586 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 8 108521868788188E 5 max 0 9999897173166792 num nulls 0 met long INT64 UNCOMPRESSED DO 0 FPO 20069536 SZ 244586 244586 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 9221656197292706803 max 9223232869150310363 num nulls 0 dim sv string BINARY UNCOMPRESSED DO 0 FPO 20314122 SZ 460339 460339 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST no stats for this column dim mv int array INT32 UNCOMPRESSED DO 0 FPO 20774461 SZ 9740584 9740584 1 00 VC 2363940 ENC RLE PLAIN ST min 2147462202 max 2147465086 num nulls 0 dim mv string array BINARY UNCOMPRESSED DO 0 FPO 30515045 SZ 70269656 70269656 1 00 VC 2373041 ENC RLE PLAIN ST no stats for this column dim sv int INT32 UNCOMPRESSED DO 0 FPO 100784701 SZ 204578 204578 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 2146586718 max 2146995692 num nulls 0 dim mv double array DOUBLE UNCOMPRESSED DO 0 FPO 100989279 SZ 19618358 19618358 1 00 VC 2416477 ENC RLE PLAIN ST min 3 2332416607383507E 6 max 0 9999963662410412 num nulls 0 dim mv float array FLOAT UNCOMPRESSED DO 0 FPO 120607637 SZ 9906684 9906684 1 00 VC 2404955 ENC RLE PLAIN ST min 4 529953E 6 max 0 9999939 num nulls 0 dim sv float FLOAT UNCOMPRESSED DO 0 FPO 130514321 SZ 204570 204570 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 1 579523E 5 max 0 99991333 num nulls 0 met int INT32 UNCOMPRESSED DO 0 FPO 130718891 SZ 204578 204578 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 2146612524 max 2147058892 num nulls 0 dim sv long INT64 UNCOMPRESSED DO 0 FPO 130923469 SZ 244586 244586 1 00 VC 93898 ENC BIT PACKED PLAIN DICTIONARY ST min 9221351713267308050 max 9223334243910011623 num nulls 0 ```

Xiang Fu · Answer

can you run `parquet tools` to see the file metadata

Xiang Fu · Answer

< kish>

kish · Answer

Hi thanks for looking

kish · Answer

Could you name your parquet file as data foo bar baz quiz 131168051 parquet and check again

kish · Answer

I have a feeling that underscores in the file name is causing issue

Xiang Fu · Answer

``` parquet tools meta tmp data foo bar baz quiz 131168051 parquet file file tmp data foo bar baz quiz 131168051 parquet creator parquet mr version 1 8 0 build 0fda28af84b9746396014ad6a415b90592a98b3b extra parquet avro schema = type record name record fields name met float type float name met double type double name dim mv long type type array items long name dim sv double type double name met long type long name dim sv string type string name dim mv int type type array items int name dim mv string type type array items string name dim sv int type int name dim mv double type type array items double name dim mv float type type array items float name dim sv float type float name met int type int name dim sv long type long file schema record met float REQUIRED FLOAT R 0 D 0 met double REQUIRED DOUBLE R 0 D 0 dim mv long REQUIRED F 1 array REPEATED INT64 R 1 D 1 dim sv double REQUIRED DOUBLE R 0 D 0 met long REQUIRED INT64 R 0 D 0 dim sv string REQUIRED BINARY O UTF8 R 0 D 0 dim mv int REQUIRED F 1 array REPEATED INT32 R 1 D 1 dim mv string REQUIRED F 1 array REPEATED BINARY O UTF8 R 1 D 1 dim sv int REQUIRED INT32 R 0 D 0 dim mv double REQUIRED F 1 array REPEATED DOUBLE R 1 D 1 dim mv float REQUIRED F 1 array REPEATED FLOAT R 1 D 1 dim sv float REQUIRED FLOAT R 0 D 0 met int REQUIRED INT32 R 0 D 0 dim sv long REQUIRED INT64 R 0 D 0 row group 1 RC 93898 TS 131168051 OFFSET 4 ```

Xiang Fu · Answer

seems no difference

Xiang Fu · Answer

I was using `ParquetRecordReaderTest java` to create a test parquet file

Xiang Fu · Answer

i 100x the records written to create row groups

Xiang Fu · Answer

a row group is like 512MB

Xiang Fu · Answer

so my data file is 1 3gb then it has 3 row groups

Xiang Fu · Answer

could you check the schema for that file

Xiang Fu · Answer

or you can try to use parquet Record reader to read the file in the test

Xiang Fu · Answer

``` Override protected RecordReader createRecordReader throws Exception ParquetRecordReader recordReader = new ParquetRecordReader recordReader init dataFile sourceFields null return recordReader ``` just like this

Xiang Fu · Answer

then try to iterate on the reader to see if there is anything being returned

kish · Answer

thanks

Xiang Fu · Answer

please let me know if you need any help on this Also if you could share a parquet file with me I could check it further on my side

kish · Answer

sure thx