This message was deleted.
# generals
Slackbot
10/13/2023, 10:00 AMThis message was deleted.
l
Laksh Singla
10/13/2023, 10:31 AMYou cannot work around it. Though row sizes being this large in the first place is unusual. Curious if you are using datasketches
Laksh Singla
10/13/2023, 10:32 AMYou can ingest via native ingestion, though large row sizes can produce other problems with that, but not gauranteed to do so.
p
pratush maheshwari
10/13/2023, 10:35 AMWe have an array column where we expect these many values to come up, we are doing the ingestion using sql, loading csv file from s3
pratush maheshwari
10/13/2023, 10:36 AMI did try to split the rows based on some threshold, like split the rows in two, if array size is more than 50k
l
Laksh Singla
10/13/2023, 10:36 AMFrame size is a hardcoded value, and you won’t be able to modify it during the runtime
p
pratush maheshwari
10/13/2023, 10:36 AMBut Can see huge performance degradation
pratush maheshwari
10/13/2023, 10:36 AMAny way we can get the default value changed?
l
Laksh Singla
10/13/2023, 10:37 AMif you build it from the source, then you can change it
p
pratush maheshwari
10/13/2023, 10:38 AMCan you elaborate this? Is it any easy config change?
l
Laksh Singla
10/13/2023, 10:38 AMbut creating a too large of a value can reduce the parallelism, so you’d need to test it out.
p
pratush maheshwari
10/13/2023, 10:39 AMI’ll test it, but can you guide me how do we increase the frame size?
Laksh Singla
10/13/2023, 10:43 AMstandard_frame_size < large_frame_size
Laksh Singla
10/13/2023, 10:44 AMAlso if you increase the size by a factor of 2, the number of processors will reduce by factor of 2. So that will also affect the performance, you’d need to adjust the heap size then
Laksh Singla
10/13/2023, 10:44 AMstandard_frame_size < large_frame_size -> But they can be the same length too
2 Views