https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
r
The files were written using pandas and passed compression as gzip: df.to_parquet(f'mapping.parquet.gz', compression='gzip')
b
that's bizarre
g
I believe this pandas 
to_parquet
call does the compression inside the parquet file, not as a regular gzip wrapper
so try saving it as 
mapping.parquet
instead of 
mapping.parquet.gz
. this is more correct since it's really parquet format (with gzip inside), not gzip format. if the extension is 
.parquet
then druid will load it using the parquet reader without trying to gunzip it first, and i tshould work
r
Thanks @Gian Merlino I will give this a try. Appreciate the feedback
g
good luck!
r
It worked, It also works changing to gzip. Pandas documentation suggests fully spelling out gzip seen below. Is this something you feel druid should handle? .gz vs .gzip is very common.
Copy code
df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
>>> df.to_parquet('df.parquet.gzip',
...               compression='gzip')  
>>> pd.read_parquet('df.parquet.gzip')
I noticed even the tutorials in documentation show .gz ...wikipedia.json.gz
g
ah, good to hear!
in druid we do gzip decompression of anything ending specifically in 
.gz
, so 
.gzip
would be treated as a non-gzip-compressed file
in terms of how druid handles it, 
.parquet
and 
.parquet.gzip
are identical (both are treated as a parquet file) — whereas 
.parquet.gz
is treated as a parquet file inside a gzip archive
r
got it. thank you for the explanation
2 Views
Open in Slack
PreviousNext
Powered by