Hi all! I would like to know if the auth system exists and I'm missing something or these feature isn't available yet... Thanks in advance!

Hi all, There is one issue that when we are adding/dropping table for sync, then all the previous data at destination is lost. Is there any way to fix this?

Hi All, I'm working on evaluating if our own database solution can be added to Airbyte as a source. I've forked the Airbyte repo and now I'm about to create a new branch so I'm wondering if there's a naming convention I should follow when it comes to branch names, etc.?

Hey, Since it is not possible to “edit” and choose what normalization version you want like you can with connectors, do you think it’d be alright to switch normalization version tags to follow airbyte core versions instead? It’d make it much easier to publish new normalization images (when releasing airbyte) instead of doing it whenever a PR is merged…. (the docker image is not useable until there is a release of airbyte anyway)

Hi! Im developing a custom connector for an API and I just added a second stream but I am unable to make the new stream appear when selecting streams to sync in the UI. I think I need som help here.

Anyone else get this error when trying to build? Seems like a gradle problem...

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-docker']
> Failed to apply plugin 'airbyte-docker'.
   > A problem occurred starting process 'command 'docker''

docker

is on my path -

$ which docker
/usr/local/bin/docker

A little more stack trace -

Caused by: org.gradle.api.internal.plugins.PluginApplicationException: Failed to apply plugin 'airbyte-docker'.
        at ...
        ... 250 more
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'docker''
        at ...
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'docker'
        at ...
Caused by: java.io.IOException: Cannot run program "docker" (in directory "/Users/me/git/airbyte/airbyte/airbyte-cli"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
        ... 8 more

Hi Everyone , Facing two issues at my end, not sure if this was answered in the thread already, i did look for it though. Airbyte version: 0.29.16-Alpha • OS Version / Instance: Linux/ AWS EC2 • Deployment: Docker • Source Connector : Zendesk/Iterable • Destination Connector : S3 1st issue - My sync job between iterable and s3 is in a loop. The log says : 2021-10-11 062055 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062105 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062115 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062125 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062135 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062145 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062155 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062205 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062215 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062225 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062235 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062245 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062255 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062305 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062315 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062325 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062335 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062345 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc 2021-10-11 062355 INFO () LogClientSingleton(setJobMdc):146 - Setting docker job mdc Its in the same loop for over an hour and then it fails. 2nd issue - For the sync job between zendesk support and s3, the data in table 'tickets' is broken. The column 'description' has multiple entire emails in it which is breaking the data when exported to csv through airbyte. The data from description ,which has over 100 of lines of string data(email data) is falling into other columns also and hence cannot use the data. There is no option of dropping this column as i dont really need it but now the entire data cannot be used. Please do advice on the workaround for both these problems.

Dear Team. I have a question on Airbyte's roadmap for performance and scalability. Potentially I believe I could try work on performance improvements, if I lucky to have some time. Could someone please comment on my spectations bellow ?: 1. Inspected Python CDK and

source-file

, as well as

source-s3

. Seems that ~60% of time source spends converting data into AirbyteMessage (before transformers) and later making

json.dump

. Are there any plans on making these conversions less painful? I would be happy to get any info to understand in which direction this architecture moves generally. 2. Are there any plans for scalability? At this moment conversions and transformations, performed in

source

container, both are obvious subject to run in parallel. For me it looks pretty perspective to have

source

responsible only for data fetch in some raw form (byte arrays?) and delegate or complex conversions, transformations/normalization to scalable middle layer (even naive Apache Spark Streaming would be good improvement I suppose). May I ask which direction does Airbyte follow to deal with scalability?

Hi! Where would I inject a string into the url for a Python API source using the Python CDK? In my case the user needs to provide both an authentication token and a numerical client id which needs to be a part of the API URL.

I am trying to use APIs(https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#auth) to create source, destination and trigger sync. How do I request/get auth token? Is it necessary?

Hi team can someone tell me how to rollback my airbyte version if I'm deployed using docker-compose and .env files? Having an issue with the current version related to nginx

Hi guys, is it possible to provide a filter while sync

Hello, I have a doubt about setting up an S3 source, is it possible to use an assumed role to connect instead of specifying the secret Id and Secret? Lets suppose my airbyte is in an EC2 instance that has all the rights to an S3 private bucket via Assumed Rule, is it actually possible to connect this way by leaving the Secret Id and Key blank ? Thanks!

Is it possible to manually overwrite the schema creation?

Facing some issues scaling Airbyte - I have gone through https://docs.airbyte.io/operator-guides/scaling-airbyte - but it is not helping - Is there any other resource I can refer to scale our Airbyte Instance? We are running the instance on a e2-highcpu-32 (32 vCPUs, 32 GB memory) VM There are about 25 connections setup similarly (syncing different databases from a same mongo instance to different dbs in the same snowflake)

Which version of Airbyte was the source search endpoint released on? I’m querying 0.30.6 and seeing a 404, but can use the connection search endpoint

Are releases that have not been tagged as “Latest release” safe to upgrade to? Is there any distinction between the “Latest release” (on Github) and the other alpha releases?

Did I miss something and do we now have M1 support?

Hi, I have a GCP instance running Airbyte, my S.O. is ubuntu, and I have a network error

401 Unauthorized

when trying to

GET

``airbyte.mydomain.io/manifest.json` . Anyone had this problem and have any idea on how to fix? Thanks in advance

How many connections can we have in airbyte? Are there inherent limitations?

Deployment: Docker Airbyte Version: v0.30.22-alpha Source name: Mysql (RDS) Destination name: Snowflake When I am trying to add namespace in snowflake, it is showing 'normalisation failed'. Can anyone please help?

Jared/Benoit, have either of you seen this Temporal error before?

Error: Operation DescribeNamespace failed.
Error Details: rpc error: code = NotFound desc = Namespace default does not exist.
{"level":"error","ts":"2021-10-26T12:07:54.656Z","msg":"Internal service error","service":"frontend","error":"Not enough hosts to serve the request","logging-call-at":"workflowHandler.go:3406","stacktrace":"<http://go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).error\n\t/temporal/service/frontend/workflowHandler.go:3406\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).StartWorkflowExecution\n\t/temporal/service/frontend/workflowHandler.go:497\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).StartWorkflowExecution.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:1105\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:116\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).StartWorkflowExecution\n\t/temporal/service/frontend/dcRedirectionHandler.go:1101\ngo.temporal.io/api/workflowservice/v1._WorkflowService_StartWorkflowExecution_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.4.0/workflowservice/v1/service.pb.go:1047\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:136\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1051\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:100\ngoogle.golang.org/grpc.chainUnaryServerInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1037\ngo.temporal.io/api/workflowservice/v1._WorkflowService_StartWorkflowExecution_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.4.0/workflowservice/v1/service.pb.go:1049\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871|go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).error\n\t/temporal/service/frontend/workflowHandler.go:3406\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).StartWorkflowExecution\n\t/temporal/service/frontend/workflowHandler.go:497\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).StartWorkflowExecution.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:1105\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:116\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).StartWorkflowExecution\n\t/temporal/service/frontend/dcRedirectionHandler.go:1101\ngo.temporal.io/api/workflowservice/v1._WorkflowService_StartWorkflowExecution_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.4.0/workflowservice/v1/service.pb.go:1047\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:136\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1051\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:100\ngoogle.golang.org/grpc.chainUnaryServerInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1037\ngo.temporal.io/api/workflowservice/v1._WorkflowService_StartWorkflowExecution_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.4.0/workflowservice/v1/service.pb.go:1049\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871>"}

The only change was changing a couple of database password hmm NVM false alarm

Chris, what is the difference between importData and importDataIntoWorkspace? Does the former assume multiple workspaces?

Hi guys. Anyone who has experience with developing Shopify auth app and connecting to airbyte ?

Hi, the docs refer to a Google Adwords connector here https://docs.airbyte.io/integrations/sources/google-adwords but it doesn't appear to be a source in the repo.

Hi All, is there a specific reason why dedup is not supported in MySQL? Any technical limitation?

Hi all, is there a way to add two new columns with some default data.. for a table generated by airbyte in postgres? this is used for downstream processing in our application.

Hi, guys! I had a conversation with @Chris (deprecated profile) regarding the step 3 from https://docs.airbyte.io/connector-development#publishing-a-connector . Does this step is still valid? The reason why I am asking the last version of bigquery-denormalized destination connector has valid version in

resources/seed/destination_definitions.yaml

and

airbyte-integrations/connectors/destination-bigquery-denormalized/Dockerfile

but not in

resources/config/STANDARD_DESTINATION_DEFINITION/079d5540-f236-4294-ba7c-ade8fd918496.json

. Thanks.

This is a general think out loud point to other Airbyters - I’m running stable now and it actually took me 3 minutes to download all the images at about 11 MB/s. I don’t remember it being this long previously. Looks like our total image footprint is now ~ 3GBs, not surprising since we’ve been adding more code and functionality. We should keep image size in mind for the future. Hopefully getting rid of the scheduler will help this a bit.

➜  ~ docker images | head -n 30 | grep 0.30.23-alpha
airbyte/webapp                       0.30.23-alpha                                           705cee2dda23   6 days ago      53MB
airbyte/server                       0.30.23-alpha                                           fee3d39ea555   6 days ago      800MB
airbyte/worker                       0.30.23-alpha                                           9f0cac3aa409   6 days ago      1.2GB
airbyte/scheduler                    0.30.23-alpha                                           786de4d6c25a   6 days ago      759MB
airbyte/db                           0.30.23-alpha                                           73344a9224f2   6 days ago      192MB

We need to read the tables generated from Airbyte in our app. So need a SQL builder to dynamically build queries to get the data.