Hi team, does anyone know the column capacity lim...
# ask-community-for-troubleshootingj
Jhon Edison Bambague Calderon
10/31/2022, 8:15 PMHi team, does anyone know the column capacity limit of the s3 source connector for a csv file. I have a csv with more than 400 columns and it generates the following error
Jhon Edison Bambague Calderon
10/31/2022, 8:15 PM Copy code
2-10-31 19:54:00 [1;31mERROR[m i.a.c.i.LineGobbler(voidCall):114 - pyarrow.lib.ArrowInvalid: CSV parse error: Empty CSV file or block: cannot infer number of columns
2022-10-31 19:54:00 [1;31mERROR[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):113 - CSV parse error: Empty CSV file or block: cannot infer number of columnsJhon Edison Bambague Calderon
10/31/2022, 8:16 PMI have the possibility to split the file, but knowing this limit would be a great help, thank you very much.
s
Saj Dider (Airbyte)
11/01/2022, 3:38 PMHey @Jhon Edison Bambague Calderon,
I don’t have an exact answer for you but I found some places in the source code that may help you understand the issue:
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/csv_parser.py#L93
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/csv_spec.py
Do you have empty columns somewhere in your csv file?
7 Views