Jarrod Parkes
09/01/2021, 4:55 PMIvan Fokeev
09/03/2021, 12:55 PMdockerRepository
in master
and that’s affecting the migration.
Could you, please, delete one of https://github.com/airbytehq/airbyte/blob/master/airbyte-config/init/src/main/reso[…]ESTINATION_DEFINITION/81740ce8-d764-4ea7-94df-16bb41de36ae.json or https://github.com/airbytehq/airbyte/blob/master/airbyte-config/init/src/main/reso[…]ESTINATION_DEFINITION/8aaf41d0-f6d2-46de-9e79-c9540f828142.json ?Zygimantas Vasiliauskas
09/06/2021, 1:54 PMDavin Chia (Airbyte)
Jarrod Parkes
09/07/2021, 2:47 PMAndrey Morskoy
09/09/2021, 12:34 PMJenny Brown
09/09/2021, 3:42 PMIvan Fokeev
09/09/2021, 6:32 PMMohammad Safari
09/09/2021, 11:16 PMDavin Chia (Airbyte)
Ivan Fokeev
09/10/2021, 12:58 PMSanjeev Singla
09/12/2021, 9:36 PMAndrey Morskoy
09/13/2021, 8:25 AMMohammad Safari
09/13/2021, 8:37 PMcreate or replace table
and creates the entire table at each sync step.
Has anybody thought of replacing this with materialized view? it is automatically supported by bigquery (so need to run it each time) and is incremental. so you only pay for new dataMohammad Safari
09/14/2021, 10:22 AMDavin Chia (Airbyte)
Andrey Morskoy
09/15/2021, 6:04 AMKubePodProcess.STATUS_CHECK_INTERVAL_MS
is 20 second, amng them POD preparation takes 14 sec.
Can I have mentioned 3 pods prepared beforehand and then reused? (Like pre-heat worker pool with queue maybe)?Sujit Sagar
09/15/2021, 10:46 AMsujit@cloudshell:~/airbyte$ SUB_BUILD=PLATFORM ./gradlew build
Building Airbyte Sub Build: PLATFORM
> Task :spotlessJava FAILED
Step 'removeUnusedImports' found problem in 'airbyte-analytics/src/main/java/io/airbyte/analytics/TrackingClientSingleton.java':
null
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.diffplug.spotless.java.GoogleJavaFormatStep$State.lambda$constructRemoveUnusedFunction$3(GoogleJavaFormatStep.java:190)
at com.diffplug.spotless.java.GoogleJavaFormatStep$State.lambda$createRemoveUnusedImportsOnly$1(GoogleJavaFormatStep.java:167)
at com.diffplug.spotless.FormatterFunc.apply(FormatterFunc.java:32)
at com.diffplug.spotless.FormatterStepImpl$Standard.format(FormatterStepImpl.java:78)
at com.diffplug.spotless.FormatterStep$Strict.format(FormatterStep.java:76)
at com.diffplug.spotless.Formatter.compute(Formatter.java:230)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:201)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:188)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.processInputFile(SpotlessTaskImpl.java:71)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.performAction(SpotlessTaskImpl.java:57)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at org.gradle.api.internal.project.taskfactory.IncrementalInputsTaskAction.doExecute(IncrementalInputsTaskAction.java:32)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.AbstractIncrementalTaskAction.execute(AbstractIncrementalTaskAction.java:25)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:555)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:56)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$run$1(DefaultBuildOperationExecutor.java:71)
at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.runWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:45)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:71)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:540)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:523)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$300(ExecuteActionsTaskExecuter.java:108)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:271)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:260)
at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$0(ExecuteStep.java:33)
at java.base/java.util.Optional.map(Optional.java:265)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)
at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:67)
at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:36)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:49)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:34)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:43)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:44)
at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:54)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':spotlessJava'.
> java.lang.reflect.InvocationTargetException
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at <https://help.gradle.org>
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>
BUILD FAILED in 2s
3 actionable tasks: 1 executed, 2 up-to-date
Dejan Antonic
09/15/2021, 2:29 PMMatthew Tovbin
09/15/2021, 8:48 PMAndrey Morskoy
09/16/2021, 6:47 AMJobScheduler.jobsScheduled
? https://github.com/airbytehq/airbyte/blob/edcd83dd1ae57ca4e33f290724f1995c2a0e53cb[…]er/app/src/main/java/io/airbyte/scheduler/app/JobScheduler.java. Seems like a minor bug for me, just askingMartin Larsson
09/16/2021, 8:20 PMvirt_env) martin@MacBook-Pro source-provet % python3 main.py read --config secrets/config.json --catalog integration_tests/catalog.json
Logg:
Traceback (most recent call last):
File "main.py", line 33, in <module>
launch(source, sys.argv[1:])
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 125, in launch
for message in source_entrypoint.run(parsed_args):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 113, in run
config_catalog = self.source.read_catalog(parsed_args.catalog)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/source.py", line 48, in read_catalog
return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))
File "pydantic/main.py", line 578, in pydantic.main.BaseModel.parse_obj
File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 3 validation errors for ConfiguredAirbyteCatalog
streams -> 0 -> stream
field required (type=value_error.missing)
streams -> 0 -> sync_mode
field required (type=value_error.missing)
streams -> 0 -> destination_sync_mode
field required (type=value_error.missing)
Davin Chia (Airbyte)
Davin Chia (Airbyte)
Sujit Sagar
09/17/2021, 2:39 AMgunu
09/17/2021, 5:22 AMMohammad Safari
09/17/2021, 7:03 PMDavin Chia (Airbyte)
(e.g. one connection, one table, 120m rows, 50gb data, mysql -> snowflake)This is what I have now. How many columns are there and what's the average size of each row?
Artem Astapenko
09/20/2021, 7:10 PMAndrey Morskoy
09/21/2021, 8:48 AMsource-file
connector, but failed in :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles
. Here is what I get when run it separately:
./gradlew :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles --info
> Task :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles FAILED
Caching disabled for task ':airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles' because:
Caching has not been enabled for the task
Task ':airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'bin/generate-protocol-files.sh''. Working directory: /projects/airbyte/airbyte-integrations/bases/airbyte-protocol Command: bin/generate-protocol-files.sh
Successfully started process 'command 'bin/generate-protocol-files.sh''
Invalid file format
:airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles (Thread[Execution worker for ':',5,main]) completed. Took 1.086 secs.