Hi, i am trying postgres to s3 parquet . *int8*(bigint) fields in postgres are converted to integer in parquet. because of that data getting corrupted. any way make this big integer in parquet?

Hi. yes it is possible though you need to write some code - very basic one. you will need to read the field and convert the dtype. You can use pandas for small amounts of data (max 10GIB of data and max 16GIB of RAM) or for higher amounts of data you can use dask or even pyspark for bigdata if needed though dask can handle big data very good.

thanks for response. in s3 parquet data it is int. so it lost. could not convert back to big int. so post processing would not work. you meant, create new connector or fork the s3 ...assume, we can not extend since it is docker or plugin in something .