Airbyte #contributing-to-airbyte

Jarrod Parkes

09/01/2021, 4:55 PM

not sure the best place to ask the question. but let’s say ive got airbyte deployed with a connection that takes an “incremental, dedup’ed source” and dumps those records into a database. are there common approaches to taking this data and synthesizing events or running comparisons against changed records? for example, via airbyte and what is present in the DB, i could detect that “some change” occurred for a record with a certain primary key. now i want to do something more useful with that information and (1) find out the nature of the exact change and (2) notify some service about the specific change

Ivan Fokeev

09/03/2021, 12:55 PM

Hi all, there are two destinations with the same

dockerRepository

master

and that’s affecting the migration. Could you, please, delete one of https://github.com/airbytehq/airbyte/blob/master/airbyte-config/init/src/main/reso[…]ESTINATION_DEFINITION/81740ce8-d764-4ea7-94df-16bb41de36ae.json or https://github.com/airbytehq/airbyte/blob/master/airbyte-config/init/src/main/reso[…]ESTINATION_DEFINITION/8aaf41d0-f6d2-46de-9e79-c9540f828142.json ?

Zygimantas Vasiliauskas

09/06/2021, 1:54 PM

Is it possible to pass incremental column value from airflow to airbyte incremental sync?

Davin Chia (Airbyte)

09/07/2021, 5:41 AM

Jared, looks like Kube Test instability is due to socat/KIND

Jarrod Parkes

09/07/2021, 2:47 PM

did i see the MongoDB destination is available 🤔 ? i didnt see it in the choices for destinations

Andrey Morskoy

09/09/2021, 12:34 PM

Dear Team. Do we have documentation on how to: • start Airbyte developemnt in IntellijIDEA (if any particularities) • debug airbyte components

Jenny Brown

09/09/2021, 3:42 PM

What’s the status on our github secrets quota issue? I need to add a couple more (environment secrets) and I’m not sure who was working on the “ran out of secrets” issue or where we’re at with it. cc @Jared Rhizor (Airbyte)

Ivan Fokeev

09/09/2021, 6:32 PM

Hi! What does this error mean? I’m getting server unresponsive sometimes

Mohammad Safari

09/09/2021, 11:16 PM

can I get permission to send a PR to airbyte? my id is mohammad-bolt

Davin Chia (Airbyte)

09/10/2021, 11:11 AM

It looks like our next release will be a minor due to the notifications contribution. Will this work well with our new Flyway migration system? (Maybe more of a question for Liren)

Ivan Fokeev

09/10/2021, 12:58 PM

Trying to migrate on production database. Does it work? Don’t understand what to do

Sanjeev Singla

09/12/2021, 9:36 PM

Hi Guys, I am using docker run commands on source image and I am new to it. Could anyone help me to know what should be the config file path in below command? docker run --rm -i <source-image-name> check --config <config-file-path>

Andrey Morskoy

09/13/2021, 8:25 AM

Dear Team. Could you please point me to correct research direction before I went to deep into the wrong one. I have simple job (read 18 lines CSV from HTTPS -> default normalization -> save to postgres). Made run in 2 modes • local docker-compose, dev run took 7 seconds • local minikube run (hot one, all images - source, dest, normalization - are pulled already) - took 78 seconds. Hotspots are source and destination worker pods workloads, ~ 30 sec each Both job logs are attached in a thread. Could you please advice, where should I look in K8S deployment to understand/debug this performance issue

Mohammad Safari

09/13/2021, 8:37 PM

bigquery suggestion. right now we use DBT for basic normalization. which does something like

Copy code

create or replace table

and creates the entire table at each sync step. Has anybody thought of replacing this with materialized view? it is automatically supported by bigquery (so need to run it each time) and is incremental. so you only pay for new data

Mohammad Safari

09/14/2021, 10:22 AM

This message was deleted.

Davin Chia (Airbyte)

09/14/2021, 10:34 AM

FYI Liren, the connector auto upgrades are working octavia rocket on Cloud

Andrey Morskoy

09/15/2021, 6:04 AM

Dear Team. If I have • some popular source (like File) • popular destination (like Postgres) • standard normalization • users with small data, but realtime or near realtime requirement • K8S run mode The best I can do now for them (after tuning)

KubePodProcess.STATUS_CHECK_INTERVAL_MS

is 20 second, amng them POD preparation takes 14 sec. Can I have mentioned 3 pods prepared beforehand and then reused? (Like pre-heat worker pool with queue maybe)?

Sujit Sagar

09/15/2021, 10:46 AM

Copy code

sujit@cloudshell:~/airbyte$ SUB_BUILD=PLATFORM ./gradlew build
Building Airbyte Sub Build: PLATFORM

> Task :spotlessJava FAILED
Step 'removeUnusedImports' found problem in 'airbyte-analytics/src/main/java/io/airbyte/analytics/TrackingClientSingleton.java':
null
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.diffplug.spotless.java.GoogleJavaFormatStep$State.lambda$constructRemoveUnusedFunction$3(GoogleJavaFormatStep.java:190)
        at com.diffplug.spotless.java.GoogleJavaFormatStep$State.lambda$createRemoveUnusedImportsOnly$1(GoogleJavaFormatStep.java:167)
        at com.diffplug.spotless.FormatterFunc.apply(FormatterFunc.java:32)
        at com.diffplug.spotless.FormatterStepImpl$Standard.format(FormatterStepImpl.java:78)
        at com.diffplug.spotless.FormatterStep$Strict.format(FormatterStep.java:76)
        at com.diffplug.spotless.Formatter.compute(Formatter.java:230)
        at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:201)
        at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:188)
        at com.diffplug.gradle.spotless.SpotlessTaskImpl.processInputFile(SpotlessTaskImpl.java:71)
        at com.diffplug.gradle.spotless.SpotlessTaskImpl.performAction(SpotlessTaskImpl.java:57)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.project.taskfactory.IncrementalInputsTaskAction.doExecute(IncrementalInputsTaskAction.java:32)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
        at org.gradle.api.internal.project.taskfactory.AbstractIncrementalTaskAction.execute(AbstractIncrementalTaskAction.java:25)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:555)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:56)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$run$1(DefaultBuildOperationExecutor.java:71)
        at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.runWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:45)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:71)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:540)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:523)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$300(ExecuteActionsTaskExecuter.java:108)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:271)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:260)
        at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$0(ExecuteStep.java:33)
        at java.base/java.util.Optional.map(Optional.java:265)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)
        at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:67)
        at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:36)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:49)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:34)
        at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:43)
        at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)
        at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)
        at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:44)
        at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:54)
      

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':spotlessJava'.
> java.lang.reflect.InvocationTargetException

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 2s
3 actionable tasks: 1 executed, 2 up-to-date

Dejan Antonic

09/15/2021, 2:29 PM

Is this your first time deploying Airbyte: Yes OS Version / Instance: Ubuntu 21.4 VM Deployment: Kubernetes Airbyte Version: 0.29.17-alpha Hi team, Currently the way Airbyte’s entry of credentials is handled is through the UI. Secrets are masked when viewing credentials but they can be downloaded from the UI. The UI has no access controls so anyone with access to the cluster could access the UI. Is there a way to disable UI or make it read only so potential customers with access to UI can't use it to download config?

Matthew Tovbin

09/15/2021, 8:48 PM

Is AirbyteApiClient for Java available somewhere on a Maven repository?

Andrey Morskoy

09/16/2021, 6:47 AM

Dear Team. Could someone explain me, where is the increment for

JobScheduler.jobsScheduled

? https://github.com/airbytehq/airbyte/blob/edcd83dd1ae57ca4e33f290724f1995c2a0e53cb[…]er/app/src/main/java/io/airbyte/scheduler/app/JobScheduler.java. Seems like a minor bug for me, just asking

Martin Larsson

09/16/2021, 8:20 PM

Hi! Im getting a validation error when trying out “read” in the terminal.

Copy code

virt_env) martin@MacBook-Pro source-provet % python3 main.py read --config secrets/config.json --catalog integration_tests/catalog.json

Logg:

Copy code

Traceback (most recent call last):
  File "main.py", line 33, in <module>
    launch(source, sys.argv[1:])
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 125, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 113, in run
    config_catalog = self.source.read_catalog(parsed_args.catalog)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/source.py", line 48, in read_catalog
    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))
  File "pydantic/main.py", line 578, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 3 validation errors for ConfiguredAirbyteCatalog
streams -> 0 -> stream
  field required (type=value_error.missing)
streams -> 0 -> sync_mode
  field required (type=value_error.missing)
streams -> 0 -> destination_sync_mode
  field required (type=value_error.missing)

Davin Chia (Airbyte)

09/17/2021, 12:45 AM

Liren, should we do a quick retro on the recent DB bug in 0.29.18? I'm wondering if there was anything more I could have done to catch it before release

Davin Chia (Airbyte)

09/17/2021, 1:33 AM

Liren, I didn't realise we were using the S3 client under the hood for the GCS destination. Is that because the blob apis are actually compatible?

Sujit Sagar

09/17/2021, 2:39 AM

How does Airbyte load the data into destination? For example, I have to load 1 month's Google Analytics data into Postgres. Would it extract all data and load everything at once (or) would it extract and load periodically?

gunu

09/17/2021, 5:22 AM

@Davin Chia (Airbyte) is there any intention for airbyte to publish this helm chart somewhere?

Mohammad Safari

09/17/2021, 7:03 PM

I want to try using postgress source with 10K fetch size as it is slow with large tables. can somebody with access help me: • create a custom branch and merge this PR into a custom branch • build a custom source-postgress image so I can try that?

Davin Chia (Airbyte)

09/20/2021, 7:24 AM

@gunu we are going to do some stress testing this week and I want to include the CDC failure case you are seeing and hopefully get more data to fix this. I was hoping for more info on the table's schema.

(e.g. one connection, one table, 120m rows, 50gb data, mysql -> snowflake)

This is what I have now. How many columns are there and what's the average size of each row?

Artem Astapenko

09/20/2021, 7:10 PM

As for storybook hosting - it is possible to bundle it into static file and host in s3 bucket ( https://storybook.js.org/docs/react/workflows/publish-storybook )

Andrey Morskoy

09/21/2021, 8:48 AM

Dear Team. I have tried to build

source-file

connector, but failed in

:airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles

. Here is what I get when run it separately:

./gradlew :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles --info

Copy code

> Task :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles FAILED
Caching disabled for task ':airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles' because:
  Caching has not been enabled for the task
Task ':airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Starting process 'command 'bin/generate-protocol-files.sh''. Working directory: /projects/airbyte/airbyte-integrations/bases/airbyte-protocol Command: bin/generate-protocol-files.sh 
Successfully started process 'command 'bin/generate-protocol-files.sh''
Invalid file format
:airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles (Thread[Execution worker for ':',5,main]) completed. Took 1.086 secs.