Hello, my Airbyte EC2 instance is running out of space, I have about 3gb left out of a 40gb disk

not sure how to clear out space or what is consuming it all

I'm also experiencing a normalization error with snowflake on a new connection:

2022-04-06 20:46:00 INFO () DefaultNormalizationWorker(run):63 - Running normalization.
2022-04-06 20:46:00 INFO () LineGobbler(voidCall):85 - Checking if airbyte/normalization:0.1.42 exists...
2022-04-06 20:46:00 INFO () LineGobbler(voidCall):85 - airbyte/normalization:0.1.42 was found locally.
2022-04-06 20:46:00 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4838/0/normalize --network host --log-driver none airbyte/normalization:0.1.42 run --integration-type snowflake --config destination_config.json --catalog destination_catalog.json
2022-04-06 20:46:00 INFO () LineGobbler(voidCall):85 - Running: transform-config --config destination_config.json --integration-type snowflake --out /data/4838/0/normalize
2022-04-06 20:46:01 INFO () LineGobbler(voidCall):85 - Namespace(config='destination_config.json', integration_type=<DestinationType.snowflake: 'snowflake'>, out='/data/4838/0/normalize')
2022-04-06 20:46:01 INFO () LineGobbler(voidCall):85 - transform_snowflake
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - Traceback (most recent call last):
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/bin/transform-config", line 8, in <module>
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -     sys.exit(main())
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 196, in main
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -     TransformConfig().run(args)
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 49, in run
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -     transformed_config = self.transform(integration_type, original_config)
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 76, in transform
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -     transformed_integration_config = {
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_config/transform.py", line 152, in transform_snowflake
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 -     "password": config["password"],
2022-04-06 20:46:01 ERROR () LineGobbler(voidCall):85 - KeyError: 'password'
2022-04-06 20:46:01 INFO () TemporalAttemptExecution(lambda$getWorkerThread$2):155 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Normalization Failed.

Hi Team, I'm trying to setup a new source with "Google Sheets" Provided Client ID, Client Secret, Spreadsheet Link but Airbyte is throwing

Unable to connect with the provided credentials to spreadsheet.

Error: Requested spreadsheet was not found.

version : 0.35.63-alpha Logs

2022-04-07 06:42:43 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/5a2be7d8-afed-41c5-8679-14d4cbc800cc/0/logs.log
2022-04-07 06:42:43 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.63-alpha
2022-04-07 06:42:43 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-google-sheets:0.2.10 exists...
2022-04-07 06:42:43 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-google-sheets:0.2.10 was found locally.
2022-04-07 06:42:43 INFO i.a.w.p.DockerProcessFactory(create):106 - Creating docker job ID: 5a2be7d8-afed-41c5-8679-14d4cbc800cc
2022-04-07 06:42:43 INFO i.a.w.p.DockerProcessFactory(create):158 - Preparing command: docker run --rm --init -i -w /data/5a2be7d8-afed-41c5-8679-14d4cbc800cc/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-sheets:0.2.10 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.35.63-alpha -e WORKER_JOB_ID=5a2be7d8-afed-41c5-8679-14d4cbc800cc airbyte/source-google-sheets:0.2.10 check --config source_config.json
2022-04-07 06:42:45 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Giving up get(...) after 1 tries (googleapiclient.errors.HttpError: <HttpError 404 when requesting <https://sheets.googleapis.com/v4/spreadsheets/1qmx4q9th1f4U2EHHxZ-mWTiVVcV_aBX9VpEHlRuXYs8?includeGridData=false&ranges=1%3A1&alt=json> returned "Requested entity was not found.". Details: "Requested entity was not found.">)
2022-04-07 06:42:45 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Formatted error: Requested spreadsheet was not found.
2022-04-07 06:42:45 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Check failed
2022-04-07 06:42:45 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...

Hi Team, I have created a connection using Jira as source & S3 (using Amazon S3) as destination. I have enabled toggles "Render Issue Fields" & "Expand Issue Changelog" in Jira configuration. & selecting streams "Issues" & "Issue fields" from fetched schema for replication. When I replicate data at my destination, these toggles are not generating any data. At S3 destination csv file has empty Rendered body column. Can anyone assist me how can I get the functioning for these toggles? does it have to do anything with the destination or file format i am using?

I have uploaded my custom dbt model on github and added url in custom tranformation. but i got this exception, Someone Help.

Hi everyone, I have a huge table to ingest in which some lines could be deleted. It's not possible for me to full refresh my table, due to it size. Do you know any solution to detect deleted lines? I think about a new column like

_airbyte_deleted_row

(i didn't find it in documentation or git issues) Thx in advance for your answer.

Is this your first time deploying Airbyte: Yes OS Version / Instance: Ubuntu 20.04 Deployment: Kubernetes Airbyte Version: 0.35.36-alpha Description: When configuring a conector from the Google Analytics to Redshift the the field name and the destination field name has a prefix 'ga' like we can see on the picture below. Is it possible to remove the 'ga' prefix from the destination field name?

Hello, I've been experiencing the following issue with a snowflake destination

I have another snowflake destination that was set up much before the new one I just created, and that one works fine

Connection to Postgres has worked but I can't fetch the schema

is there an official support channel?

I've submitted an issue on github: https://github.com/airbytehq/airbyte/issues/11814

Hi all. I'm trying out airbyte using the docker repo and trying to land some data into my Azure storage account. I'm getting the following error when trying to set up the destination. I've confirmed the storage account is accessible from a different computer on the same network, so the storage account should be accessible. Not sure how else to troubleshoot this issue. Caused by: java.net.UnknownHostException: failed to resolve 'XXXXX.blob.core.windows.net' after 2 queries

Hi there! Thank you for your time posting in our Slack Channel. This week we’re moving our support discussions over to discuss.airbyte.io; if you would be so kind as to post your questions there, you’ll be able ask for help from fellow users as well as the Airbyte team. It is our intention that as of Friday afternoon (Pacific Time) we will lock this channel to everything except administrator posts. We’ll leave everything in place for the rest of April if you need to get information or troubleshooting from existing threads, with the intention of archiving the channel in early May. We really appreciate everyone helping us to build our community, and we’re looking forward to the next phase.

Hi there! Thank you for your time posting in our Slack Channel. This week we’re moving our support discussions over to discuss.airbyte.io; if you would be so kind as to post your questions there, you’ll be able ask for help from fellow users as well as the Airbyte team. It is our intention that as of Friday afternoon (Pacific Time) we will lock this channel to everything except administrator posts. We’ll leave everything in place for the rest of April if you need to get information or troubleshooting from existing threads, with the intention of archiving the channel in early May. We really appreciate everyone helping us to build our community, and we’re looking forward to the next phase.

Hi Everyone, I have several sheets in Office 365 Sharepoint and Onedrive to ingest and it seems like there is no connector for Ms O365. Is there a workaround? I desperately want to integrate O365 data sources to my warehouse. Thank you in advance for your guidance.

Hey everyone. I'm currently trying to use Airbyte to grab google analytics events data from BigQuery and dump it into Databricks. The reason I'm doing this is because this gives more more granular data than the default GA connector, which just uses the API to pull different reports. Right now, I'm getting errors in the Airbyte connector, which I learned is because my table in BigQuery is using nested arrays in the table (a

record

in BigQuery language). I tested this theory with a table that didn't have nested arrays and it worked fine. So, I know the problem, but turning the raw GA data into a flat table is no small feat. So, here are my questions: 1. Does Airbyte support ingesting data from BigQuery that is semi-structured? 2. If not, anybody know how to help me flatten out this table in BigQuery?

lse

Hi there I meet problems when I want to create a shopify source , it shows 504 error.

Hello Everyone, I am injesting from MongoDB to BQ which has been successful. But I updated the schema based on new changes and it keeps failing after I run the sync again. Attached image shows the last log messages. Please how do I resolve this?

Hello, how do I prevent my instance from running out of space?

I have an EC2 instance with 40Gb, the instance is exclusively used for Airbyte, it has no space left on the device

Any ideas why the upgrade from 0.29.15-alpha to 0.32.0-alpha-patch-1 would not work?

I've run

<https://raw.githubusercontent.com/airbytehq/airbyte/v0.32.0-alpha-patch-1/{.env,docker-compose.yaml}>

, but then I am unable to connect to the server once I do

docker-compose up

, I get connection refused

and if I do the same thing with master, I get

Cannot upgrade from version 0.29.15-alpha to version 0.35.65-alpha directly. First you must upgrade to version 0.32.0-alpha. After that upgrade is complete, you may upgrade to version 0.35.65-alpha

Any ideas?

Hello guys, anyone could explain to me how airbyte state works? I've been facing some errors like a first sync is running but fails and the next attempt starts from beginning. The state should have been written and next attempt starts from them? I'm using Postgres to S3 connectors.

Hello, we have a created a custom source for our requirement. We've been facing issues in adding that to local UI. Could you please help us figure out a way to add that to the local UI? See below screenshots for my error: