Hey everyone, hope you are well. I am facing an interesting issue where i am trying to insert data from parquet files using files() function. now the issue is that one of the column contains vector embeddings and each row is approximately 37MB i am marking this row is as STRING NOT NULL. Somehow the data is being converted and failing and NULLs are returned. Therefore i am getting null in not null column error.
Any help to fix this issue is greatly appreciated, is there any internal hard limit on string columns? One more thing i am also using this column in order by clause in the DDL