airbytehq #contributing-to-airbyte

Hi there 👋 Super happy to open my first Airbyte PR octavia partying I just wanted to share some minor fixes we did in our instance on the Airbyte's Helm Chart, and also tried to propose an easy way to manage logs on S3. I'm not really used to helm template functions, maybe I did some stuff too verbose or missed something, don't hesitate to comment ✍️ Thanks for the great product and doc octavia loves

Harsha Teja Kanna

12/07/2021, 5:23 PM

https://github.com/temporalio/sdk-java/releases/tag/v1.6.0

Manav

12/08/2021, 10:12 PM

Hey team! we’re seeing almost all of our syncs fail with the following error:

io.airbyte.workers.WorkerException: Error while getting checking connection

- has anyone run into this before?

Mohamed Magdy

12/09/2021, 5:09 PM

Hello Airbytians, is the right channel to RFR https://github.com/airbytehq/airbyte/pull/8675?

Davin Chia (Airbyte)

12/10/2021, 3:24 PM

Hey team, we’ve been seeing build failures for builds extending longer than 15 mins due to degraded performance for self hosted runners. This started yesterday, and was reported by the general community about 10 hours ago. Github just became aware of this about 30 mins ago. Follow here for the action: https://github.com/actions/runner/issues/1546 Status page: https://www.githubstatus.com/ I too lost a couple of hours today scratching my head until I found the issue. Sorry for the trouble!

Matthew Tovbin

12/10/2021, 6:25 PM

FYI CVE-2021-44228 - https://www.randori.com/blog/cve-2021-44228/

Jared Rhizor (Airbyte)

12/10/2021, 6:37 PM

Head's up! We're starting to move our Java development onto Java 17 today. If you're using a Java version manager like jabba/jenv it should be easy to change your configuration. Otherwise on Mac you should just run brew install openjdk@17
to get to the most recent version. If you see a version error when running Gradle commands, try running

./gradlew clean

or removing

.gradle

from the root directory or Airbyte. If you're switching between

master

and a current branch you may have to do this multiple times (merging latest

master

into your branch after the Java 17 update should prevent the need for running into this multiple times). To configure IntelliJ, check to make sure that the in

Project Structure

the

Project SDK

is Java 17 and the language level is set to 17. This may happen automatically, but it likely depends on your configuration. Please let me know if you see anything surprising locally or on CI. We're first releasing this for our internal cloud project and then will soon release it for the OSS project.

Rheza

12/12/2021, 2:19 PM

Hi Dev Team, I am new to Airbyte, I have a question for the team, does Airbyte support RBAC with SAML / SSO integration?

Oleksandr Tsukanov [GL]

12/13/2021, 9:13 AM

Hi, guys! Faced some issues during migration on Java 17 on Mac OS:

Copy code

airbyte-worker     | Exception in thread "main" io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998952187s. 
airbyte-worker     |    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
airbyte-worker     |    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
airbyte-worker     |    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
airbyte-worker     |    at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282)
airbyte-worker     |    at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61)
airbyte-worker     |    at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:275)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:182)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41)
airbyte-worker     |    at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:40)
airbyte-worker     |    at io.airbyte.workers.WorkerApp.main(WorkerApp.java:189)

What was done by me: • Switched global env to JDK 17 • Rebuild docker images • Switched IntelliJ to JDK 17 • run

VERSION=dev docker-compose up

Did somebody faced some similar issue?

Bruno

12/13/2021, 2:41 PM

Hi everyone! We are developing a tool that downloads Airbyte's data (sources, connections and destinations mainly), to store it inside a Git repo for future use This tool allows then to use this information (in json format) to load and restore (or configure a new Airbyte instance) as it was at "extraction time". It's quite like the Export-Import stuff inside Settings in the UI We acknowledge certain fields are hidden behind ****** in the API response (we use Git Secrets to load them when needed), but some others (like S3) are exposing critical information, i.e

aws_secret_access_key

Is there anywhere in the docs, or Airbyte's repos, where we could find a list of the hidden (and non-hidden) fields for the different connectors? Finding them out by trial and error, taking into account the vast number of connectors available, is extremely difficult.

Mohamed Magdy

12/13/2021, 11:04 PM

I am on 0.33.11 now and getting this

Copy code

021-12-13 22:57:58 INFO i.a.s.RequestLogger(filter):95 - {cloud_workspace_app_root=/workspace/server/logs} - REQ 10.30.33.154 GET 200 /api/v1/health
======= service endpoint: <http://airbyte-minio>:%!s(int=9000)

Ryan N

12/14/2021, 1:08 AM

maybe the better question is how to set up this parameter of .env under kube/overlays/stable-with-resource-limit

Copy code

# Worker pod tolerations and node selectors
JOB_POD_TOLERATIONS=
JOB_POD_NODE_SELECTORS=

Eugene Krall

12/14/2021, 9:55 AM

Hi. How can I find the dbt model which was used for default normalization? I need to make a minor tweaks but don't know much about DBT so I though if I look at the model that what actually used, it would be easier for me to just make those minor tweaks without writing everything from scratch

Sai

12/14/2021, 3:09 PM

Hi, Could someone point me to the source or a quick brief about how the data is transferred between Source and Destination and once the pipeline is triggered, i.e If I use

Salesforce

as the source and

GCP GCS

as a destination, do airbyte pull all the data from the source first then stage it on temporary storage and then start streaming the temporarily stored data to the destination i.e

GCP GCS

?? I see

streams

usage but would like to understand how this is handled with large datasets like 20 GB or 30 GB data. Disclaimer : Just started using Airbyte.

Cristiano Sarmento

12/14/2021, 7:19 PM

Hi, is there any special reason why there is no connector with Twitter? Thanks!

gunu

12/14/2021, 11:59 PM

i have CDC connector. i would like to add a table without performing a full reset. i know that you cant do that via the UI and implications from the pointer on the binlogs. but what are the steps to doing this manually. i was thinking: 1. create new connector with specific table to perform a full sync (connector B) 2. add table to existing connector without resetting (connector A) 3. remove connector B 4. if i have incremental dedupe on all tables, i am assuming this should be fine following subsequent syncs?

Manav

12/15/2021, 3:58 AM

Does anyone have experience writing logs for EC2 deploys to S3? Our workers seem to be generating a lot of data, and EBS storage is filling up quickly

Jason Edwards

12/17/2021, 11:14 PM

Hi, I found an issue with the Postgres source connector 0.3.14+, would the best thing to do be create an issue/bug report on github? Specifically, the issue seems related to this change that excludes the tables where the DB user doesn’t have select privileges on the table. In my case, I have some materialized views, which the DB user does have select privileges on, but the materialized views don’t show up in the catalog in Airbyte.

Alexander Furer

12/20/2021, 9:51 AM

Parametrized source/destination connector , Is it supported ?

Eugene Krall

12/21/2021, 8:02 PM

Not sure but looks like deduped history doesn't work as expected when using custom normalizations. My undestanding is that the deduplications gets overwritten by my custom dbt? source - MongoDB, destination - BQ

Eugene Krall

12/22/2021, 2:27 PM

I've exported Airbyte normalization dbt models, edited them a little, pushed them to github repository and am trying to make them run during sync. getting this error

Database Error in model messages (models/airbyte_incremental/whatsapp/messages.sql)

Syntax error: Expected "(" or keyword SELECT or keyword WITH but got keyword CREATE at [9:5]

The model is trying create an intermediary table like it did in the original models

Remi Salmon

12/22/2021, 8:07 PM

Hi, not sure where to post this but there is a bug in the Google Ads connector where it is not able to normalize the keyword_report table (log error:

100038 (22018): Numeric value 'KEYWORD' is not recognized

) because of a wrong type definition here: https://github.com/airbytehq/airbyte/blob/13ac480a8b0024360d35c20fd6d640296b57f137[…]source-google-ads/source_google_ads/schemas/keyword_report.json

developersteve

12/23/2021, 1:59 AM

Just wondering if theres a security channel or anyone looking at the security side of airbyte, Ive done a scan from the public repo using snyk free tier (Disclaimer: i do work for them) and it looks like theres some possible known open source vulns and some container config issues to be looked at. Possible code issues too, but those need exploring some more to see if they are false positives. continued in thread...

Seth Saperstein

12/25/2021, 5:21 AM

Hey everyone! Wasn’t entirely sure where to post this as it’s related to configuration and integration. I’m looking to use DBT for normalization into Redshift. I’d love for all downstream models of my raw data to be ran anytime the airbyte job runs. This is possible with the

dbt run --select <model>+

however to get the raw model to my dbt project, I don’t love the suggestion of hopping into the airbyte container, determining the normalization directory, and grabbing the generated dbt model, syncing that back to a dbt repo, and then integrating the dbt repo on the airbyte job configuration. Has anyone found a better way of exporting dbt models? I’m also planning on running on dbt cloud and this configuration means that I cannot “deploy” models via dbt cloud when the source dataset changes. Alternatively, I could trigger the dbt cloud api but that isn’t possible directly via airbyte, which means I would then have to use Airflow to schedule the airbyte job and then kick of the dbt cloud job. This means that Airflow code must be written for new data sources and I’m looking to keep data integration and normalization self-service to speed up development time for new datasets. If anyone has suggestions I’m all ears.

Zawar khan

12/27/2021, 1:11 AM

Hi Everyone, I have a problem that, when selecting Raw data - no normalization, data get inserted as json blob but with basic normalization I get error. ERROR: Database Error Normalization Fails. I am new to airbyte can some one help. As json blob data get inserted in form of rows like: eg row 1: {key:value} row 2: {key:value} row 3: {key:value} I would like to have keys as column names. I am using postgres.

Wisse Jelgersma

12/28/2021, 12:22 PM

Hi All, I am creating a connector for a system that can also list the deleted rows from a certain point in time. Is there a way to mark rows as deleted in an incremental stream? Would it work if I add ab_cdc_deleted_at and ab_cdc_updated_at columns to my output?

Fernando Nava

12/28/2021, 9:55 PM

Looking into using cloudsql postgres source and wondering if pgoutput is more performant/stable than wal2json ?