Hi Team When I process a huge data in google sheets and get Airbyte #announcements

Hi Team, When I process a huge data in google shee...

Nithin Chandran

01/20/2021, 12:32 PM

Hi Team, When I process a huge data in google sheets and get below error in api call. Your inputs are really appreciated here

Copy code

2021-01-20 05:54:56 ERROR (/tmp/workspace/42/1) LineGobbler(voidCall):69 - googleapiclient.errors.HttpError: <HttpError 400 when requesting <https://sheets.googleapis.com/v4/spreadsheets/1IixkLcgI78cbBBpVcAwFP6BRI5C4sXZ6P2CRPzbs7t0/values:batchGet?ranges=Mass%21151154%3A151354&majorDimension=ROWS&alt=json> returned "Range (Mass!151154:151354) exceeds grid limits. Max rows: 151000, max columns: 26". Details: "Range (Mass!151154:151354) exceeds grid limits. Max rows: 151000, max columns: 26">
2021-01-20 05:54:56 DEBUG (/tmp/workspace/42/1) DefaultAirbyteSource(close):109 - Closing tap process

Nithin Chandran

01/20/2021, 1:42 PM

I have records as max row limit to 151000

Thomas

01/20/2021, 1:43 PM

Are you reading from a spreadsheet or writing to a spreadsheet?

Nithin Chandran

01/20/2021, 2:00 PM

@Thomas: Reading from google spreadsheet and writing into snowflake table

Thomas

01/20/2021, 2:13 PM

@Nithin Chandran how many rows does your spreadsheet has?

Thomas

01/20/2021, 2:13 PM

can you add an empty row on the bottom of the spreadsheet?

Nithin Chandran

01/20/2021, 2:18 PM

Sure

Thomas

01/20/2021, 2:34 PM

does that work?

Nithin Chandran

01/20/2021, 2:34 PM

@Thomas: Same error after made some empty rows in bottom of spreadsheet

Copy code

ERROR (/tmp/workspace/43/0) LineGobbler(voidCall):69 - googleapiclient.errors.HttpError: <HttpError 400 when requesting <https://sheets.googleapis.com/v4/spreadsheets/1IixkLcgI78cbBBpVcAwFP6BRI5C4sXZ6P2CRPzbs7t0/values:batchGet?ranges=Mass%21151154%3A151354&majorDimension=ROWS&alt=json> returned "Range (Mass!151154:151354) exceeds grid limits. Max rows: 151000, max columns: 26". Details: "Range (Mass!151154:151354) exceeds grid limits. Max rows: 151000, max columns: 26">

Thomas

01/20/2021, 3:17 PM

Google sheets API is weird sometimes 😄

01/20/2021, 3:28 PM

Hi Nithin, thanks for reporting this issue. How many rows/columns does your sheet have?

Nithin Chandran

01/20/2021, 3:50 PM

@s: 150996 rows and 26 columns

01/20/2021, 3:54 PM

OK I think I see the issue here — I believe this is a bug with the connector where it’s trying to get more rows than the sheet contains.

01/20/2021, 3:56 PM

what version of airbyte are you using?

01/20/2021, 3:57 PM

you can

cat .env

to see

Nithin Chandran

01/20/2021, 4:34 PM

0.11.2 -- version

Nithin Chandran

01/20/2021, 4:34 PM

VERSION=0.11.2-alpha DATABASE_USER=docker DATABASE_PASSWORD=docker DATABASE_DB=airbyte CONFIG_ROOT=/data WORKSPACE_ROOT=/tmp/workspace DATA_DOCKER_MOUNT=airbyte_data DB_DOCKER_MOUNT=airbyte_db WORKSPACE_DOCKER_MOUNT=airbyte_workspace # todo (cgardens) - when we are mount raw directories instead of named volumes, *_DOCKER_MOUNT must # be the same as *_ROOT. # Issue: https://github.com/airbytehq/airbyte/issues/578 LOCAL_ROOT=/tmp/airbyte_local LOCAL_DOCKER_MOUNT=/tmp/airbyte_local TRACKING_STRATEGY=segment # todo (cgardens) - hack to handle behavior change in docker compose. *_PARENT directories MUST # already exist on the host filesystem and MUST be parents of *_ROOT. # Issue: https://github.com/airbytehq/airbyte/issues/577 HACK_LOCAL_ROOT_PARENT=/tmp BUILD_TAG=dev

Nithin Chandran

01/20/2021, 4:35 PM

@s: Shall I delete the source connection and recreate and check?

01/20/2021, 4:36 PM

@Nithin Chandran can you try going to the admin panel and changing the google sheets connector version to 0.1.5?

01/20/2021, 4:36 PM

(unless it is already there)

Nithin Chandran

01/20/2021, 4:38 PM

Want to change tag to 0.1.5?

01/20/2021, 4:38 PM

yes

01/20/2021, 4:38 PM

then retry the sync

Nithin Chandran

01/20/2021, 4:39 PM

Sure

Nithin Chandran

01/20/2021, 5:05 PM

@s: I see different error, is there any limitation in google api's and json?

Copy code

ERROR (/tmp/workspace/48/0) LineGobbler(voidCall):69 - googleapiclient.errors.HttpError: <HttpError 429 when requesting <https://sheets.googleapis.com/v4/spreadsheets/1IixkLcgI78cbBBpVcAwFP6BRI5C4sXZ6P2CRPzbs7t0/values:batchGet?ranges=Mass%21110954%3A111154&majorDimension=ROWS&alt=json> returned "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute per user' of service '<http://sheets.googleapis.com|sheets.googleapis.com>' for consumer 'project_number:529482092061'.". Details: "[{'@type': '<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>', 'reason': 'RATE_LIMIT_EXCEEDED', 'domain': '<http://googleapis.com|googleapis.com>', 'metadata': {'quota_limit': 'ReadRequestsPerMinutePerUser', 'quota_metric': '<http://sheets.googleapis.com/read_requests|sheets.googleapis.com/read_requests>', 'service': '<http://sheets.googleapis.com|sheets.googleapis.com>', 'consumer': 'projects/529482092061'}}]">

Nithin Chandran

01/21/2021, 3:32 PM

@s: Any help on above error?

01/21/2021, 3:38 PM

@Nithin Chandran apologies - missed your last message from yesterday. We’re actively working on a fix to this issue right now and expect to have a fix soon

01/21/2021, 3:39 PM

The second issue you are seeing is because rate limit handling was introduced in 0.1.6

01/21/2021, 3:39 PM

we’ll push a new version with the fix for the original issue and I think you should be set after

Nithin Chandran

01/21/2021, 3:42 PM

@s: Thank you. Please keep me posted when new version will be available for use

01/21/2021, 10:46 PM

@Nithin Chandran can you upgrade your google sheets connector to version 0.1.7 from the Admin page and try again? we just pushed a fix for this issue: https://github.com/airbytehq/airbyte/pull/1762/files

Nithin Chandran

01/22/2021, 12:43 PM

@s: Sure . I will try now

Nithin Chandran

01/22/2021, 2:30 PM

@s: Now worked. Thank you so much

01/22/2021, 3:36 PM

Amazing!

👏 1

4 Views

Open in Slack

Previous Next