Airbyte #ask-community-for-troubleshooting

Ajay Kulkarni

02/01/2023, 8:42 PM

kubectl logs airbyte-webapp-5b7b558b5-g7mdd -n airbyte /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2023/02/01 203906 [emerg] 1#1: host not found in upstream "dev-airbyte.sbloantest.com:8001" in /etc/nginx/conf.d/default.conf:2 nginx: [emerg] host not found in upstream "dev-airbyte.sbloantest.com:8001" in /etc/nginx/conf.d/default.conf:2

Nag R

02/01/2023, 8:58 PM

Hi , I am trying to get a list of jobs using api .... I know I have been running sync jobs ... but when I hit the api , I am not getting any job information : I am getting this response: <html> <head> <title>405 Not Allowed</title> </head> <script language="javascript" \> window.TRACKING_STRATEGY = "segment"; window.AIRBYTE_VERSION = "0.40.32"; window.API_URL = "/api/v1/"; window.CONNECTOR_BUILDER_API_URL = "/connector-builder-api"; </script> <body> <center> <h1>405 Not Allowed</h1> </center> <hr> <center>nginx/1.23.3</center> </body> </html>

Raymond Chow

02/01/2023, 9:09 PM

Hi, it's first time trying to get airbyte to work/setup. I've created a local docker instance of airbyte and I am using and trying to setup

Copy code

Airbyte Version: 0.39.42-alpha
Source: MongoDb (0.1.15)
Destination: Snowflake (0.4.34)

The issue I ran into is I believe my snowflake connection is dropping mid process or not connecting properly. I've ran test on the connection, via settings, on both the source and destination and they were successful. I'm not too sure what might be the issue, I see that there's a DEADLINE_EXCEEDED in the logs and it seems that it was midway of reading records for a replication task the destination cancelled. When I check the activity of on snowflake it seems to be creating and dropping

AIRBYTE_CONNECTION_TEST

stages. I am using OpenVPN because mongoDB requires a VPN connection to access if that matters.

Nag R

02/01/2023, 9:13 PM

what is the configid in airbyte api for getting the list of jobs ? "configTypes": [ "check_connection_source" ], "configId": "1", "includingJobId": 0, "pagination": { "pageSize": 0, "rowOffset": 0 } }

Rafael Paraíso Rossim

02/01/2023, 10:13 PM

hello, I use Airbyte open-source and I need to connect ClickUp, could someone help me please??

Francisco Viera

02/01/2023, 10:29 PM

i need this pull request https://github.com/airbytehq/airbyte/pull/22114 who can help me?

Matt Palmer

02/01/2023, 10:41 PM

Is there any support planned for

GEOMETRY

types? The current method of syncing data does not handle these types well and frequently results in silent errors and dropped rows

Ulan Yisaev

02/01/2023, 10:45 PM

Hi team! Is there any workaround to deploy Airbyte to Azure Container Instances (ACI)? For now, it seems not possible, since a running Docker daemon (i.e.

/var/run/docker.sock

) is required to run Airbyte. And apparently Microsoft does not support DIND (docker-in-docker) in ACI for security reasons: https://github.com/MicrosoftDocs/azure-docs/issues/89486 https://github.com/MicrosoftDocs/azure-docs/issues/43947

Kyle Pounder

02/01/2023, 11:08 PM

random feedback, but i'm surprised there's no place in the top nav on airbyte.com to log in to airbyte cloud... embarrassingly i had to hunt around for where to log back in:

👍🏼 1

Satish Srinivasan

02/01/2023, 11:23 PM

hi all. I am new to airbyte. just downloaded it today and created a simple config based custom connector and deployed it, created a source and tested ok. I was able to run the read command locally and see data. However, when I create a connection from my custom source to s3 target I get this error.

Copy code

2023-02-01 23:00:06 ERROR i.a.s.e.UncaughtExceptionHandler(handle):28 - Uncaught exception
java.lang.IllegalArgumentException: json schema nodes should always be object nodes. path: [FieldNameOrList{fieldName='type', isList=false}] actual: "string"
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:177) ~[io.airbyte-airbyte-commons-0.40.32.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:205) ~[io.airbyte-airbyte-commons-0.40.32.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchema(JsonSchemas.java:103) ~[io.airbyte-airbyte-commons-0.40.32.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaWithFilteredCollector(JsonSchemas.java:138) ~[io.airbyte-airbyte-commons-0.40.32.jar:?]
	at io.airbyte.commons.json.JsonSchemas.collectPathsThatMeetCondition(JsonSchemas.java:153) ~[io.airbyte-airbyte-commons-0.40.32.jar:?]
	at io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.maskAllSecrets(JsonSecretsProcessor.java:71) ~[io.airbyte.airbyte-config-config-persistence-0.40.32.jar:?]
	at io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.prepareSecretsForOutput(JsonSecretsProcessor.java:59) ~[io.airbyte.airbyte-config-config-persistence-0.40.32.jar:?]

Chen Lin

02/01/2023, 11:29 PM

Hi everyone, is gzip format supported for s3 source? I don't see it in the source_definition_specifications for

S3 Source Spec

Lior Chen

02/02/2023, 12:18 AM

Hey All, I’m experiencing problems with the shopify connector. it basically loads all the orders and customers of the shop to memory. eventually server crashes on OOM. is there any resolution?

Roberto Tolosa

02/02/2023, 2:31 AM

hi! for the streaming connectors on the roadmap for the future, we were curious – will these streaming connectors make use of Snowpipe if the destination is Snowflake? thinking about data warehouse cost implications if the streaming connector won't use snowpipe (and whether we should build our own source->staging->snowpipe sync)

Roberto Tolosa

02/02/2023, 2:44 AM

we're looking into triggering a workflow based on whether Salesforce data that we sync into Snowflake has changed. should i assume that if we set up the SFDC->Snowflake sync to incremental deduped, it should use/support CDC (and thus support tracking using table streams)?

Prashant Barge

02/02/2023, 6:27 AM

Hello all, I have just cloned the airbyte's repo from github and started it in docker ... This is the error which i am getting. Can you please let me know how i can resolve this error = "POST /api/v1/workspaces/list HTTP/1.1" 502 497 "http://143.110.240.150:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"

김건희

02/02/2023, 6:31 AM

Hello everyone! I'm using airbyte on eks. Now I want to connect to s3 using arnawsiam made in aws console. arn have s3 full access policy, i add it to serviceaccount anntation. but i reponse access denied error.

Copy code

ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):163 - Exception attempting to access the S3 bucket:
Stack Trace: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

how could i success this way?

Andreas Gustafsson

02/02/2023, 6:38 AM

Hi! Is it possible to trigger syncs with Prefect 2 when using the Airbyte Cloud? When researching it seems like it's only possible when hosting Airbyte yourself

➕ 2

Mark Nuttall-Smith

02/02/2023, 8:38 AM

Hi, question about the Airtable source connector (or maybe source connectors generally). Is it possible to always refresh the schema/reset the streams before every sync, so the destination always reacts to changes upstream?

Ismail Tigrek

02/02/2023, 11:44 AM

Where can I get help about the API?

Claudio Cavallo

02/02/2023, 12:55 PM

morning guys, any know how to solve this issue:

could not write lock file "postmaster.pid": No space left on device

Revital Eres

02/02/2023, 1:21 PM

Hi I am running normalization using the following docker command and I appreciate help in understanding why the workdir is

/data/1/0/normalize

(I do not see it in mysql.Dockerfile). Also, the files written to the local dir (

/tmp/airbyte_local

) are with root permission so I can not delete them. is there a way to create them with not root permission? Thanks!

Copy code

docker run  --init -i -w /data/1/0/normalize  --log-driver none  --network host -v /tmp/airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.40.14 airbyte/normalization-mysql:0.2.25 run --catalog /local/tmp58_rero0 --config /local/tmp4jbbpmox --integration-type mysql

Michael Yee

02/02/2023, 1:47 PM

For people running airbyte kubernetes, what size VM did you find works the best for syncs from dB to say s3 or snowflake? Also, can you share your helm chart/configurations or just used defaults?

Luan Araldi

02/02/2023, 2:13 PM

Hey Everyone! I'm developing a custom source connector in python, and I am having some trouble when trying to create a connection using it, I can use it to create a source successfully, but when creating a connection with it (destination to S3) Airbyte throws an unknow error, just like the image attached, I tried to validate the source using the

Copy code

docker run --rm airbyte/source-example-python:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-example-python:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-example-python:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-example-python:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json

and it works fine like this One thing I noticed is this error on the browser console when the error appears on screen

Copy code

index-01f874f4.js:39 TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at zl (SchemaError-3025fefa.js:72:6583)
    at bo (index-01f874f4.js:37:52767)
    at to (index-01f874f4.js:37:62191)
    at ra (index-01f874f4.js:37:61528)
    at di (index-01f874f4.js:37:61122)
    at wa (index-01f874f4.js:41:11784)
    at ya (index-01f874f4.js:41:932)
    at kc (index-01f874f4.js:41:863)
    at cn (index-01f874f4.js:41:722)

Krisjan Oldekamp

02/02/2023, 3:23 PM

Currently I'm in the process of setting up Airbyte using the Helm charts / Kubernetes (on GKE). Only thing I'm not sure about, Airbyte uses a lot of storage for it's connectors, how do I make sure I have enough storage allocated? Do I need to mount an additional volume (and if so, to which service/pod) or are there other settings in the values.yml file?

luisa zuluaga

02/02/2023, 3:48 PM

Hi I'm trying to set up a FTP connection with SSH Private Key, however not sure how add the SSH Key on the authentication field really appreciate your help in advance

Dusty Shapiro

02/02/2023, 4:09 PM

Is it possible to change the repository and tag of the

busybox

Docker image that gets pulled when a K8s pod starts up to run a sync? We’re experiencing rate limiting from Dockerhub, so I’d like to start pushing those images up to ECR and reference them. We’re deploying on K8s via Helm Chart Thanks

Imane

02/02/2023, 4:14 PM

Hey guys, I am trying to extend the

source-pinterest

connector with the following metrics (in the

utils.py

Copy code

"VIDEO_P0_COMBINED_1,VIDEO_P100_COMPLETE_1,VIDEO_P25_COMBINED_1,VIDEO_P50_COMBINED_1,"
 "VIDEO_P75_COMBINED_1,VIDEO_P95_COMBINED_1,VIDEO_3SEC_VIEWS_1,VIDEO_MRC_VIEWS_1,"
 "APP_INSTALL_COST_PER_ACTION,CLICKTHROUGH_2,CLICKTHROUGH_1,ENGAGEMENT_2,ENGAGEMENT_1,TOTAL_CLICK_ADD_TO_CART"

but I am getting the error attached, does anyone have any idea about this?

pinterest.json

Jakub Wszolek

02/02/2023, 4:55 PM

hey, I have submitted a pull request to introduce support for Podman Compose in Airbyte, allowing it to be run in rootless environments. Please take a look: https://github.com/airbytehq/airbyte/pull/22323.

Gilberto S.S

02/02/2023, 5:15 PM

Where to find the directory airbyte_local, after run \\wsl& on the directory bar, the result take to only services, after it there is no more directory, because i want to copy a csv file to airbyte_local to set it as a source on airbyte.

Krzysztof Sikora

02/02/2023, 5:54 PM

Hi, While upgrading airbyte in our GKE we’re stumbling across an issue where

airbyte-server

and

airbyte-cron

are both trying to mount

airbyte-volume-configs

workloads are on different nodes and are failing to both attach the volume