Chris Conradi
02/07/2022, 3:41 PMEmily Cogsdill
02/07/2022, 5:05 PMEmily Cogsdill
02/07/2022, 6:21 PMRoman
02/07/2022, 10:09 PMB
with a child stream A
. A
always is called first, bypassing all the incremental and caching benefitsChristopher Wu
02/08/2022, 12:19 AMChristopher Wu
02/08/2022, 3:14 AMJackson Clarke
02/08/2022, 6:33 AMNarender Kumar
02/08/2022, 12:08 PMRaj
02/08/2022, 3:10 PMImane
02/08/2022, 4:35 PMImane
02/09/2022, 4:47 AMNarender Kumar
02/09/2022, 5:15 AMairbyte-bootloader.tar not found
.
How can i make airbyte-bootloader.tar.
How to make other dependencies(tar files) for airbyte images used in docker-compose.yml.Titas Skrebė
02/09/2022, 3:23 PM_airbyte_data
instead of putting into the respective column names? I know that it expands into columns when I enable Basic Normalization, but I would like to know if it doesn't expand by default without normalization.Jay Bujala
02/09/2022, 4:01 PMFAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':tools:code-generator:airbyteDocker'.
> A problem occurred starting process 'command 'C:\git\airbyte\tools\bin\build_image.sh''
* Try:
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':tools:code-generator:airbyteDocker'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:145)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:143)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:131)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:402)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:389)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:382)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:368)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'C:\git\airbyte\tools\bin\build_image.sh''
at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'C:\git\airbyte\tools\bin\build_image.sh'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WindowsProcessLauncher.start(WindowsProcessLauncher.java:22)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
... 4 more
Caused by: java.io.IOException: Cannot run program "C:\git\airbyte\tools\bin\build_image.sh" (in directory "C:\git\airbyte\tools\code-generator"): CreateProcess error=193, %1 is not a valid Win32 application
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 8 more
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
... 9 more
* Get more help at <https://help.gradle.org|https://help.gradle.org>
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See <https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings|https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings>
BUILD FAILED in 17s
22 actionable tasks: 11 executed, 4 from cache, 7 up-to-date
PS C:\git\airbyte>
Will Sargent
02/09/2022, 8:23 PMYiyang (Heap.io)
02/09/2022, 10:12 PM{"type": "LOG", "log": {"level": "ERROR", "message": "Encountered an exception while reading stream SourceRippling\nTraceback (most recent call last):\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 108, in read\n internal_config=internal_config,\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 141, in _read_stream\n for record in record_iterator:\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 213, in _read_full_refresh\n for record in records:\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 356, in read_records\n next_page_token = self.next_page_token(response)\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/source_rippling/streams.py\", line 23, in next_page_token\n if bool(decoded_response.get(\"has_more\", \"False\")) and decoded_response.get(\"data\", []):\nAttributeError: 'list' object has no attribute 'get'"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing SourceRippling"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceRippling runtimes:\n"}}
{"type": "LOG", "log": {"level": "FATAL", "message": "'list' object has no attribute 'get'\nTraceback (most recent call last):\n File \"main.py\", line 13, in <module>\n launch(source, sys.argv[1:])\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 116, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 107, in run\n for message in generator:\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 112, in read\n raise e\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 108, in read\n internal_config=internal_config,\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 141, in _read_stream\n for record in record_iterator:\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 213, in _read_full_refresh\n for record in records:\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 356, in read_records\n next_page_token = self.next_page_token(response)\n File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/source_rippling/streams.py\", line 23, in next_page_token\n if bool(decoded_response.get(\"has_more\", \"False\")) and decoded_response.get(\"data\", []):\nAttributeError: 'list' object has no attribute 'get'"}}
and fatal message that “list” object has no attribute “get”, but this appears right after all the records are read and yielded as output. Another endpoint I made a stream for worked fine without having this fatal message, and the only real difference I could see between the two outputs was that this one was in dictionary format enclosed in curly brackets {}, while the one that was erroring out had a response in list format with [] brackets. To fix this, I tried adding an if case to the parse_response method in the base stream that would yield each row in a for loop if the response type was a list. This gave the same result with all the records yielded, and still getting the fatal list object has not attribute message at the end. Is this a known issue by any chance, or has anyone encountered something similar?Marissa Pagador
02/10/2022, 5:04 AMNarender Kumar
02/10/2022, 5:18 PMrefresh_token
, and that this is required if I want to use the OAuth2Authenticator
class in the CDK. My source doesn’t have a refresh token.
In my check()
function, I can do a custom POST request and authenticate no problem.
Is there an option to either:
1. Use the OAuth2Authenticator class without a refresh-token somehow.
2. Just do a custom Post in my streams function instead of using the OAuth2Authenticator (like I do in check()
)
3. Is there just an entirely different way to do this
4. …or am I out of luck and Airbyte simply doesn’t support Oauth2 without refresh tokens?Toma Bernotaitė
02/10/2022, 8:13 PMToma Bernotaitė
02/10/2022, 8:51 PMZak Keener
02/10/2022, 9:58 PMZak Keener
02/11/2022, 2:01 AMpython main.py check --config sample_files/config.json
The error message (screenshot #2)
The json context in sample_files/config.json
(screenshot #3)
What interesting is if I put the same json context (e.g. "pokemon_name": "pikachu") into secrets/config.json
(I know it's not where it belongs but just for a testing), and the Check testing will run successfully (screenshot #4).
Not sure if it has anything to do with Anaconda, but other than that I'm running out of idea of the root causes.
Any support will be much appreciated!! Thanks in advance and let me know if any more info is needed.🙇♂️will
02/11/2022, 10:29 AMHưng Lê
02/11/2022, 3:04 PMdef next_page_token
I am able to build a list of different filter params that are well under the limits.
But, here is my issue:
• As soon as I run my main.py read
script, it kicks off the request on the endpoint. (this query doesn’t work, as it tries to return all the 2 million rows).
• I need this stream to wait for the next_page_token to make a few secondary requests - to build a list of filters - before starting to pull from the dataset.
What is the best way to do this?Greg Blackman
02/11/2022, 6:19 PM"createdAt": { "type": ["null", "string"], "format": "date-time" "format_string": "dd-mm-YYY HH:MM"}
Antonio Grass
02/13/2022, 1:10 AMERROR: (gcloud.beta.compute.ssh) Could not fetch resource:
all the timeEnrico Tuvera Jr.
02/14/2022, 4:59 AMEnrico Tuvera Jr.
02/14/2022, 11:54 AMIn the thread.
Tomas Balciunas
02/14/2022, 5:19 PMTomas Balciunas
02/14/2022, 8:51 PMpip install --prefix=/install .
gives an error:
error: could not create '/install': Read-only file system
. Seems like the error is happening because it's trying to install at the root file level /
macos won't let you install anything there but it was generated like this so I think I'm missing something.