This message was deleted.
# generals
Slackbot
03/07/2023, 3:41 PMThis message was deleted.
g
Gian Merlino
03/07/2023, 6:52 PMwhere are you reading your data from, what format is it in, & are you using batch or stream ingest?
v
Vadim
03/07/2023, 8:33 PMCan you post a screenshot of where you are setting the unreadable data?
m
Mes
03/08/2023, 9:29 AMThe ingestion is of type index parallel and here's copy paste of the ioConfig. It can "read" the file, but its definitely not in a human-readable format. (The file itself is fine for sure). I also tried changing the suffix of the file to just json and hoping that druid would recognize the compression. Same thing happens with zst compressed orc file (tried suffixing the file with zst and orc).
"type": "index_parallel",
"inputSource": {
"type": "s3",
"objects": [
{
"bucket": "bucket",
"path": "path/to/file.json.snappy"
}
]
},
"inputFormat": {
"type": "json",
"compression": "snappy"
}
g
Gian Merlino
03/08/2023, 6:17 PMfor batch ingest, Druid uses the suffix of the file to determine how to decompress
Gian Merlino
03/08/2023, 6:18 PMfor snappy framing format (https://github.com/google/snappy/blob/main/framing_format.txt) Druid expects
.sz<http://file.json.sz|file.json.sz>Gian Merlino
03/08/2023, 6:19 PMnote that there are multiple snappy compression formats out there. Druid supports just this one ☝️ so if it doesn't work for you, it's possible you are using one of the other formats. The most common are unframed (raw) compression, and the snappy frame format defined by Hadoop, neither of which Druid supports reading directly
m
Mes
03/08/2023, 6:57 PMThanks for the info. I'll try tomorrow and hopefully get it to work.
Mes
03/09/2023, 12:59 PMFor file.orc.zst I get
Error: Failed to sample data: java.io.IOException: Decompression error: Unknown frame descriptor
And for file.json.sz I get
Error: Failed to sample data: java.io.IOException: Not a framed Snappy stream
So I guess the snappy compression is one of the not supported group. However, with the zstd I'd like the guess the same, but this error is different. Any ideas?
Mes
03/09/2023, 2:49 PMThe file is for sure zst compressed, but there are maybe also some subsets of zst that druid supports?
Mes
03/09/2023, 2:52 PMSame compression is used for CSV and we can successfully read those. I've checked and I did include orc-extension for Druid
g
Gian Merlino
03/09/2023, 11:43 PMhmm i'm not super familiar with zst so i am not sure if there are multiple formats
Gian Merlino
03/09/2023, 11:43 PMi wonder if it's a zst problem or orc problem
Gian Merlino
03/09/2023, 11:44 PMhttps://github.com/luben/zstd-jni/issues/103 seems to indicate it's a zst problem
Gian Merlino
03/09/2023, 11:44 PMhttps://github.com/luben/zstd-jni/ is what we're using btw