Jamil B
04/06/2022, 8:31 PMJamil B
04/06/2022, 8:31 PMJamil B
04/06/2022, 8:49 PM2022-04-06 20:46:00 INFO () DefaultNormalizationWorker(run):63 - Running normalization.
2022-04-06 20:46:00 INFO () LineGobbler(voidCall):85 - Checking if airbyte/normalization:0.1.42 exists...
2022-04-06 20:46:00 INFO () LineGobbler(voidCall):85 - airbyte/normalization:0.1.42 was found locally.
2022-04-06 20:46:00 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4838/0/normalize --network host --log-driver none airbyte/normalization:0.1.42 run --integration-type snowflake --config destination_config.json --catalog destination_catalog.json
2022-04-06 20:46:00 INFO () LineGobbler(voidCall):85 - Running: transform-config --config destination_config.json --integration-type snowflake --out /data/4838/0/normalize
2022-04-06 20:46:01 INFO () LineGobbler(voidCall):85 - Namespace(config='destination_config.json', integration_type=<DestinationType.snowflake: 'snowflake'>, out='/data/4838/0/normalize')
2022-04-06 20:46:01 INFO () LineGobbler(voidCall):85 - transform_snowflake
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - Traceback (most recent call last):
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - File "/usr/local/bin/transform-config", line 8, in <module>
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - sys.exit(main())
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 196, in main
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - TransformConfig().run(args)
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 49, in run
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - transformed_config = self.transform(integration_type, original_config)
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 76, in transform
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - transformed_integration_config = {
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 152, in transform_snowflake
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - "password": config["password"],
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - KeyError: 'password'
2022-04-06 20:46:01 INFO () TemporalAttemptExecution(lambda$getWorkerThread$2):155 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Normalization Failed.
Dinesh Agrawal
04/07/2022, 6:48 AMUnable to connect with the provided credentials to spreadsheet.
Error: Requested spreadsheet was not found.
version : 0.35.63-alpha
Logs
2022-04-07 06:42:43 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/5a2be7d8-afed-41c5-8679-14d4cbc800cc/0/logs.log
2022-04-07 06:42:43 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.63-alpha
2022-04-07 06:42:43 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-google-sheets:0.2.10 exists...
2022-04-07 06:42:43 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-google-sheets:0.2.10 was found locally.
2022-04-07 06:42:43 INFO i.a.w.p.DockerProcessFactory(create):106 - Creating docker job ID: 5a2be7d8-afed-41c5-8679-14d4cbc800cc
2022-04-07 06:42:43 INFO i.a.w.p.DockerProcessFactory(create):158 - Preparing command: docker run --rm --init -i -w /data/5a2be7d8-afed-41c5-8679-14d4cbc800cc/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-sheets:0.2.10 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.35.63-alpha -e WORKER_JOB_ID=5a2be7d8-afed-41c5-8679-14d4cbc800cc airbyte/source-google-sheets:0.2.10 check --config source_config.json
2022-04-07 06:42:45 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Giving up get(...) after 1 tries (googleapiclient.errors.HttpError: <HttpError 404 when requesting <https://sheets.googleapis.com/v4/spreadsheets/1qmx4q9th1f4U2EHHxZ-mWTiVVcV_aBX9VpEHlRuXYs8?includeGridData=false&ranges=1%3A1&alt=json> returned "Requested entity was not found.". Details: "Requested entity was not found.">)
2022-04-07 06:42:45 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Formatted error: Requested spreadsheet was not found.
2022-04-07 06:42:45 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Check failed
2022-04-07 06:42:45 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
Mahak Urooj
04/07/2022, 6:59 AMAli Yousaf
04/07/2022, 8:26 AMFrederic Tapin
04/07/2022, 1:10 PM_airbyte_deleted_row
(i didn't find it in documentation or git issues)
Thx in advance for your answer.Carlos Silva
04/07/2022, 2:18 PMJamil B
04/07/2022, 4:37 PMJamil B
04/07/2022, 4:43 PMGeorge Pearse
04/07/2022, 4:45 PMJamil B
04/07/2022, 5:21 PMJamil B
04/07/2022, 6:03 PMl4ov3l3fmuV8
04/07/2022, 6:14 PMMarcos Marx (Airbyte)
04/07/2022, 6:34 PMMarcos Marx (Airbyte)
04/07/2022, 6:37 PMSuntx
04/07/2022, 9:04 PMJon Boone
04/08/2022, 12:51 AMrecord
in BigQuery language). I tested this theory with a table that didn't have nested arrays and it worked fine. So, I know the problem, but turning the raw GA data into a flat table is no small feat. So, here are my questions:
1. Does Airbyte support ingesting data from BigQuery that is semi-structured?
2. If not, anybody know how to help me flatten out this table in BigQuery?Shubhank Vijayvergiya
04/08/2022, 1:50 AMOng Rio
04/08/2022, 4:01 AMLevana Dai
04/08/2022, 9:23 AMDavid Effiong
04/08/2022, 12:07 PMJamil B
04/08/2022, 12:47 PMJamil B
04/08/2022, 12:48 PMJamil B
04/08/2022, 1:58 PMJamil B
04/08/2022, 1:59 PM<https://raw.githubusercontent.com/airbytehq/airbyte/v0.32.0-alpha-patch-1/{.env,docker-compose.yaml}>
, but then I am unable to connect to the server once I do docker-compose up
, I get connection refusedJamil B
04/08/2022, 2:00 PMCannot upgrade from version 0.29.15-alpha to version 0.35.65-alpha directly. First you must upgrade to version 0.32.0-alpha. After that upgrade is complete, you may upgrade to version 0.35.65-alpha
Jamil B
04/08/2022, 2:01 PMLeonardo de Almeida
04/08/2022, 2:52 PMRishika
04/08/2022, 5:43 PM