Hi I created a new source for tap-exacttarget based on Singer. This is the PR https://github.com/airbytehq/airbyte/pull/4175 Let me know what needs to be done in order to be able to merge it.

@Jared Rhizor (Airbyte) @Subodh (Airbyte) is running into a strange error here - the acceptance test aren't able to find the PVC even though its created earlier on. Have ya seen this before?

hey everyone, I just merged a PR which moves seed from

airbyte-config

to

airbyte-server/src/main/resources/seed

. How does that impact you? If you are building a new connector (source or destination) or making changes to an existing connector, in order to update the seed or add new entry to list of seeds, you will find the

destination_definitions.yaml

and

source_definitions.yaml

in the new path i.e

airbyte-server/src/main/resources/seed

P.S. Apology if this caused conflict for your branch/work

Improvement suggestion: Can something please be added to the table in the admin menu to sort alphabetically? There are quite a large amount of connectors in there now and it will only become more difficult to find things as more connectors are added. Thanks

I am seeing

DEADLINE_EXCEEDED

errors even if I launch Airbyte from scratch. Based on previous threads related to this issue, is that because my laptop does not have enough resource? I did only have 8GB of memory.

Hey, I'm curious about ECS based Airbyte deployments. Is this something that we can expect soon now that the k8s deployment is out? or is this lower down in priority? https://docs.airbyte.io/deploying-airbyte/on-aws-ecs

Hey Airbyte Team, two changes I think everyone should be aware of: 1. We are no longer publishing dev images. We used to publish dev images as part of the master build. This was confusing as it resulted in two potential sources of truth for

dev

images and lead to cases where code used the remote dev image instead of the local dev image. I've also manually delete the images from Dockerhub. Since no one is using this, we will stop publishing them. Lmk if anyone has concerns. 2. Log levels are now exposed as an env var, which means we can start inserting debug statements as we develop code. Should make it easier to debug user issues,

Hey, I am running on master branch with commit

0a11d88b0ba6bc6c5f00dd8257fa40ed463b6cdc

: Does anybody run into these NPE error with

./gradlew :airbyte-tests:automaticMigrationAcceptanceTest

? But if I run the command:

MIGRATION_TEST_VERSION=0.26.4-alpha ./gradlew :airbyte-tests:automaticMigrationAcceptanceTest

, then it passes? cc @Subodh (Airbyte)

When creating a workspace with the API, can you skip creating a workspace and setting preferences? The screen appears to show up regardless of whether a workspace exists or not

There is a reason why choosing the 1k batch size for JDBC databases? Could we increase this value to 10k or 50k whats the pro/cons? or, doing this value configurable? @Hudson change the default value to 50k and was able to transfer 18M rows using Mysql in 10 min. Using the default value 1k batch size the sync took 50 min. The Github issue to discuss about it: https://github.com/airbytehq/airbyte/issues/4314

Hello, Airbyte team! I want use custom dbt transformation and also want to use my server URL instead of github repo URL. I'm gonna fork my github dbt repo, clone into server and will use server URL instead of github URL in airbyte UI. Is it possible?

I tried to upgrade from 0.26.2-alpha to 0.27.0-alpha, and this time, my upgrade script ran without a hitch! I ran it like this:

./upgrade-airbyte 0.26.2-alpha 0.27.0-alpha 0.27.0-alpha

Is there any reason why we request the

/health

endpoint every 10 seconds? Like restarting services if something failed?

2021-07-01 19:55:01 INFO i.a.s.RequestLogger(filter):109 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.29.0.6 GET 200 /api/v1/health

Also, should this endpoint return more information besides {"db": "true"}

For those working on Kube, I totally recommend using

stern

(

brew install stern

) to tail the Kube logs. The features I love are 1) ability to select deployments based on regex (e.g.

stern "airbyte-s"

gives the airbyte-server and airbyte-scheduler) 2) logs are tagged with colored pod names, making it easy to read

Hi, just a simple doubt, if I want to do some customizations in the front-end (HTML / React) of the application, do I need to build with

./gradlew clean build

I'm trying to publish the Zendesk Support connector. CI is failing in the task buildSrcbuild (there are reference to other two connectors zendesk-talk and zoom-singer...? Any ideas?

FAILURE: Build failed with an exception.


* Where:
Build file '/home/runner/work/airbyte/airbyte/airbyte-integrations/connectors/source-zendesk-talk/build.gradle' line: 2

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-python']
> Failed to apply plugin 'airbyte-python'.
   > A problem occurred configuring project ':airbyte-integrations:connectors:source-zoom-singer'.
      > Could not open cp_proj generic class cache for build file '/home/runner/work/airbyte/airbyte/airbyte-integrations/connectors/source-zoom-singer/build.gradle' (/home/runner/.gradle/caches/6.7.1/scripts/4ffd4lk4g399iqzx3xoc30xgr).
         > java.lang.StackOverflowError (no error message)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

is there any stable version of the webapp code that already builds properly, or some option I can pass during the built that ignores the errors?

Hey guys, I have been wanting to add Azure Synapse as destination. I have made some changes to https://github.com/airbytehq/airbyte/blob/d382d1bb96b21fe18b33fff3f9ac6b0155ca05ce[…]airbyte/integrations/destination/mssql/SqlServerOperations.java

Hello, I’m trying to make requests with a custom header to the airbyte-server. I added the new header name I want to pass in my requests to the CorsFilter, ACCESS_CONTROL_ALLOW_HEADERS, but it seems to have no effect, is there any other place I have to change something? (I’m not familiar with Java). After the changes, I’m building the server with “./gradlew airbyte-server:build”. Thanks!

I have FE e2e tests failing with next error from time to time.

* Where:
Build file '/home/runner/work/airbyte/airbyte/airbyte-integrations/connectors/source-zendesk-talk/build.gradle' line: 2

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-python']
> Failed to apply plugin 'airbyte-python'.
   > A problem occurred configuring project ':airbyte-integrations:connectors:source-zoom-singer'.
      > Could not open cp_proj generic class cache for build file '/home/runner/work/airbyte/airbyte/airbyte-integrations/connectors/source-zoom-singer/build.gradle' (/home/runner/.gradle/caches/6.7.1/scripts/4ffd4lk4g399iqzx3xoc30xgr).
         > java.lang.StackOverflowError (no error message)

Is it a known issue?

Heads up Airbyte team, I've figured out a short term fix for the build failures we've been seeing. TLDR: • The test/publish flows will also run on ec2 instances on AMI images that contain all the dependencies required to build Airbyte. See https://github.com/airbytehq/airbyte/pull/4558/files for more context. Lmk if anyone sees issues/has questions! • The frontend builds will also run on the ec2 runner • In short we are fully on the ec2 runners now - should also see a mild boost in speeds!

How much Memory/CPU are we giving source and destination pods?

I am working on a PR in an unrelated area and got a PR build failure on kubernetes end to end tests, with

Exception in thread "main" java.lang.RuntimeException: could not find workspace with id: 5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6

I haven’t touched workspace ids; all I did was pull in the latest code. Any idea what this is coming from?

@Chris (deprecated profile) @Jared Rhizor (Airbyte) did we double check that our Java connectors are actually using the entire RAM allocated to them in the Kube resources? Java's default behaviour is to use 1/4 of the host's container memory so I'm not sure the Java connector allows itself to use all the allocated memory

The spotlessJava step is failing to auto-format my SQL file locally, yet the PR build is complaining about whitespace changes in the file. How can I bring in the PR build’s fixed version of that file, or otherwise fix my formatting?

I need to test SchedulerApp with my local code changes. What’s the best way to launch that thing, such that it’s possible to tell whether I broke it? This is my first time trying to test it in isolation rather than just running the entire docker compose setup. Or if

docker-compose up

is enough for running locally changed code, how do I find the relevant logs through the console spam?

Heads up / FYI - We’ve used a docker container for terraform in the past, for purposes of controlling the version of terraform that runs. There is a wrapper for this which gives better flexibility while maintaining version management, called

tfenv

. On mac, you can get it with

brew install tfenv

- it looks in the current directory for a

.terraform-version

file and automatically pulls the right version when you run

terraform plan

etc. This way the command line stays standard, and versions are per-directory, which gives us the ability to support a different version for a contributed connector if necessary. I’ve used tfenv in my previous work. It elegantly solves the versioning issue while enabling flexibility and growth. I’m picking up @s’s PR stubbing out terraform for ssh tunnel work, and we’ll be switching that to recommend the use of tfenv rather than the docker container, for version consistency. Feedback welcome if you see potential issues with this.

Hi Team, After starting airbyte on EC2 instance, we did some security testing. Largely looked good. However, we are curious about the curl request involving the URL app.papercups.io when we open the airbyte instance. Details below. Can any please help me understand this curl request?

curl -i -s -k -X $'POST' \
    -H $'Host: <http://app.papercups.io|app.papercups.io>' -H $'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0' -H $'Accept: */*' -H $'Accept-Language: en-US,en;q=0.5' -H $'Accept-Encoding: gzip, deflate' -H $'Content-Type: application/json' -H $'Content-Length: 456' -H $'Origin: <https://airbytelocalinstance.com>' -H $'DNT: 1' -H $'Connection: close' -H $'Referer: <https://airbytelocalinstance.com/>' \
    --data-binary $'{\"browser_session\":{\"account_id\":\"74560291-451e-7teb-a802-56706ece528b\",\"customer_id\":\"844059db-be56-490a-8c4a-f323b3e\",\"started_at\":\"2021-07-12T05:08:58.117Z\",\"metadata\":{\"os\":\"Linux\",\"browser\":\"Firefox\",\"time_zone\":\"Asia\",\"current_url\":\"<https://airbytelocalinstance.com/>\",\"host\":\"<http://airbytelocalinstance.com|airbytelocalinstance.com>\",\"pathname\":\"/\",\"browser_version\":89,\"screen_height\":900,\"screen_width\":1600,\"lib\":\"web\",\"insert_id\":\"2ssjm0kucch0swby\",\"time\":1626066538.117}}}' \
    $'<https://app.papercups.io/api/browser_sessions>'

Hi Team, Can someone guide me with how to understand the code base on Airbyte. i am new to open source community and want some directions. PS : Obviously not expecting spoon feed.

do i need to learn spring boot or some framework if i need to understand the java codes in Airbyte? Can someone help?