Getting this error in download phase of async stre...
# ask-ai
p
Getting this error in download phase of async stream. The download location is a google storage and file type is CSV `nternal Server Error: com.fasterxml.jackson.databind.JsonMappingException: String value length (20054016) exceeds the maximum allowed (20000000, from
StreamReadConstraints.getMaxStringLength()
) (through reference chain: io.airbyte.connectorbuilderserver.api.client.model.generated.StreamRead["slices"]->java.util.ArrayList[0]->io.airbyte.connectorbuilderserver.api.client.model.generated.StreamReadSlicesInner["pages"]->java.util.ArrayList[0]->io.airbyte.connectorbuilderserver.api.client.model.generated.StreamReadSlicesInnerPagesInner["response"]->io.airbyte.connectorbuilderserver.api.client.model.generated.HttpResponse["body"])`
k
The error message you are seeing:
Copy code
com.fasterxml.jackson.databind.JsonMappingException: String value length (20054016) exceeds the maximum allowed (20000000, from StreamReadConstraints.getMaxStringLength())
indicates that the response body being read during the download phase of your asynchronous stream is larger than the maximum allowed string length for deserialization in the Airbyte Connector Builder backend. This is a backend limitation, not a misconfiguration in your connector. Key points from the knowledge sources: - The Connector Builder's Download tab expects to process the downloaded data (e.g., parse CSV, JSON, etc.) and not just pass the file through to the destination. There is no documented way to configure the builder to simply download and store the file without reading and parsing it first. The builder is designed to extract records from the file, not to act as a file transfer tool [Download Tab documentation](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/async-streams#download-tab). - The error is related to the backend's deserialization limits, as seen in similar issues with large payloads and JSON mapping exceptions [GitHub Issue: destination-gcs](https://github.com/airbytehq/airbyte/issues/51592). What you can do: - If your use case is to simply transfer files (e.g., from Google Cloud Storage to a destination) without parsing them, the Connector Builder is not designed for this workflow. - If you need to process large files, you may need to use a custom connector or a different approach that can handle large file streaming and parsing, as the Connector Builder UI currently has limitations on the maximum response size it can process. Summary: The error is due to a backend string length constraint in the Connector Builder. There is no supported way in the Connector Builder UI to bypass parsing and just download/store the file as-is. For large files or file transfer use cases, consider building a custom connector or using a different tool better suited for large file handling. If you need further assistance or want to request this as a feature, you may want to open an issue or comment on related GitHub issues for visibility [Connector Builder not able to read .gz file in download tab](https://github.com/airbytehq/airbyte/issues/69765).
p
@kapa.ai will a custom CSV decoder work in this issue?
k
A message from kapa.ai
Based on the provided knowledge sources, a custom CSV decoder can be implemented using the CustomDecoder component in Airbyte’s config-based connector framework. This allows you to define your own decoding logic by specifying a custom class that extends the Decoder base class [source](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/CustomDecoder). However, the knowledge sources do not explicitly state that using a custom CSV decoder will bypass or resolve the backend string length constraint (the StreamReadConstraints.getMaxStringLength() error) encountered during the download phase. The error is related to the backend’s deserialization limits, not the decoder logic itself. Therefore, even with a custom decoder, if the backend still attempts to load the entire response body into memory as a string, the same limitation may apply. In summary: You can implement a custom CSV decoder, but the knowledge sources do not confirm that this will resolve the string length constraint issue you are experiencing. The limitation appears to be at the backend deserialization layer, not within the decoder logic itself.