This message was deleted.
# generals
Slackbot
05/02/2023, 10:36 PMThis message was deleted.
s
shraddha jain
05/03/2023, 12:10 AMI believe it supports zstd compression as mentioned here https://druid.apache.org/docs/latest/ingestion/native-batch.html#supported-compression-formats. Not sure if it is supported in 0.19 though.
shraddha jain
05/03/2023, 12:21 AMHow do we set the property to accept zstd compressed parquet data in druid server? What is the property and where do we set it?
c
Charles Smith
05/03/2023, 2:32 AMI see it in the supported list for 0.20.0: https://druid.apache.org/docs/0.20.0/ingestion/native-batch.html#compression-formats-supported
Charles Smith
05/03/2023, 2:32 AMI don't see a property where you specify the incoming compression
g
Gian Merlino
05/03/2023, 4:21 AMIIRC for Parquet decompression is automatic as part of the reader
s
shraddha jain
05/03/2023, 8:38 PMThanks Charles and Gian.
shraddha jain
05/03/2023, 8:39 PMI have some follow-up questions:
1. What property is needed to be set on druid ingestion spec to ingest zstd compressed parquet data on druid version 0.19?
2. Do we need to install any packages on druid server for druid to parse the zstd compressed parquet data?
g
Gian Merlino
05/03/2023, 8:57 PMunfortunately I don't know the answers since I haven't use parquet with zstd before 😅
Gian Merlino
05/03/2023, 8:58 PMAlso, Druid version 0.19 is really old 🙂
Gian Merlino
05/03/2023, 8:58 PMOne path here could be to add a zstd Parquet file to https://github.com/apache/druid/blob/master/extensions-core/parquet-extensions/example/compat
And add a test case in
CompatParquetReaderTestGian Merlino
05/03/2023, 8:59 PMThis'll validate it works in the latest version, and in case there is some reason it doesn't, it would provide a test case to help figure out a fix
s
shraddha jain
05/03/2023, 8:59 PMNo problem. Maybe someone else from the community might know the answer. Yeah we are moving to 0.25 in a new cluster, but for now we are maintaining 0.19.
shraddha jain
05/03/2023, 9:07 PMif the above way you mentioned can be done, that would be nice. Do we need to open a pull request for the same?
g
Gian Merlino
05/03/2023, 9:33 PMIf you do; it'd be greatly appreciated and would help ensure it always works going forward!
Gian Merlino
05/03/2023, 9:33 PMmore test cases are always a good thing
Gian Merlino
05/03/2023, 9:34 PMIf you do, post the PR link here and I'll have a look
s
shraddha jain
05/03/2023, 9:39 PMok, I mostly look at the infra side, I am not that comfortable with it. Let me check with other team member if he knows how to do it.
shraddha jain
05/03/2023, 9:42 PM@dontu balu will you be able to try it out?