<@U04SE163WC8> - I'm trying to extract data from a...
# ask-ai
c
@kapa.ai - I'm trying to extract data from a relatively large BigQuery table, but am receiving the following message. Can you provide more information? java.lang.RuntimeException: java.lang.RuntimeException: com.google.cloud.bigquery.BigQueryException: Response too large to return. Consider specifying a destination table in your job configuration.
u
This error message is related to a known issue when using Airbyte to extract data from large BigQuery tables. The error occurs because the query result is too large for BigQuery to return directly. According to the information provided in [this GitHub issue](https://github.com/airbytehq/airbyte/issues/6533), this error can occur when trying to sync large amounts of data from BigQuery (as a source) to another destination. The error message suggests that the response from BigQuery is too large to return in a single query result. BigQuery has limitations on the size of query results it can return. When a query's results exceed this limit, BigQuery suggests specifying a destination table in the job configuration to store the results instead of returning them directly. To address this issue, you have a few potential options: 1. Consider using a destination table: As suggested in the error message, you could modify your BigQuery job configuration to specify a destination table for the query results. This would allow BigQuery to write the large result set to a table instead of trying to return it directly. 2. Add a LIMIT clause: If you don't need all the data, you could try adding a LIMIT clause to your query to reduce the amount of data returned. 3. Remove ORDER BY clauses: If your query includes ORDER BY clauses, removing them might help, as mentioned in the GitHub issue: "Adding a LIMIT clause can sometimes help, or removing the ORDER BY clause." 4. Split the data: You might need to consider splitting your data extraction into smaller chunks or multiple queries to avoid hitting the response size limit. 5. Use column selection: As suggested in [another related issue](https://github.com/airbytehq/airbyte/issues/36303), you could use Airbyte's column selection feature to only sync the columns you're interested in, which would reduce the query size. If these suggestions don't resolve the issue, you may need to reach out to the Airbyte community or support for more specific guidance on handling large BigQuery extractions.
c
@Chris Potter did you manage to get a solve for this?
c
@Chirag Goel - No, we ended up going another route (unfortunately).