Airbyte #connector-development

Chris Conradi

02/07/2022, 3:41 PM

I am developing my first source connector - so far loving it! But, I need to do an OAuth2.0 authentication, and haven’t found any good documentation source for this. I basically need to get a bearer token using client_id and client_secret, and use the bearer token for any consecutive queries. Bearer token needs to be refreshed every hour. Does anybody have a good source of documentation to get an Airbyte newb going on setting up OAuth2.0 with the source connector?

Emily Cogsdill

02/07/2022, 5:05 PM

Hi team! I have a question about polling for job status for a custom destination connector. My team is developing a custom destination connector (using Python), sending data using POST requests. The API we are calling with this connector performs data import asynchronously - the POST request returns a job id, and you can then run GET requests to check the status of that job. Ideally we would like the Airbyte sync to only display “Succeeded” if the status of the job indicates it was successful. Sometimes this might take a while (up to an hour) for very long jobs. Is there a recommended approach to implementing status polling when posting data to an API with this behavior? Of course the easy thing is to use some “while” logic with a timeout to keep checking the status of the job id, but we are wondering if there is some better practice we should be aiming to use instead.

Emily Cogsdill

02/07/2022, 6:21 PM

Hi everyone! I'm curious is it possible to develop connectors locally without all repository checkout, e.g. using dependencies instead of source code? Or share your local development best practice😉 Thank you!

Roman

02/07/2022, 10:09 PM

Is there a way to order the stream reads? Seems like my streams are read alphabetically. The issue I have is that I have a parent stream called

with a child stream

always is called first, bypassing all the incremental and caching benefits

Christopher Wu

02/08/2022, 12:19 AM

Hello team! Is it possible to allow users to enter an arbitrary json object for certain connector configuration parameters without requiring them to stringify it? If I specify an object type for a parameter in the spec.json without specifying explicit properties, it doesn’t show up in the UI

Christopher Wu

02/08/2022, 3:14 AM

Hello! I have set up a connection to run every hour from the UI. How can I control the hour offset at which it runs? For example, if I wanted it to start at 5 minutes past the hour every hour. Happy to set this in a manual way (i.e. in database or via API) if it is necessary as the timing of this job is important.

Jackson Clarke

02/08/2022, 6:33 AM

Hello Team, Following is the message being thrown though cursor field was set Can I ignore the message that Unknown Cursor state/Status, How can I fix it ? Please suggest.

Narender Kumar

02/08/2022, 12:08 PM

Hello, any leads/examples on how we can implement an source FTP connector ?

Raj

02/08/2022, 3:10 PM

Hello team! Can someone help me understand how incremental streams interplay with child sub streams, please? Specifically, I’m using the Lever Hiring connector where there’s an incremental stream called Opportunities. Opportunities has multiple child streams, which stream slices of opportunities at a full refresh. How can I make those child streams also incremental?

Imane

02/08/2022, 4:35 PM

Hey guys! I am developing a connector for DV360 in Python, and the data can only be fetched in CSV format. Any ideas how to convert it to JSON efficiently to be useful for Airbyte?

Imane

02/09/2022, 4:47 AM

Hello !! I am trying to sync the data to Oracle DB using Oracle connector. 1. But I do not see Incremental - Deduped option under sync modes. As per the Oracle DB notes, it was supported. Am I missing some thing here ? Please suggest. 2. Also normalization is not supported for Oracle DB. How can one enable normalization to create normalized tables as per the notes.

Narender Kumar

02/09/2022, 5:15 AM

Hi , I made a PR for Kafka Connector, Adding SCRAM-SHA-512 auth. https://github.com/airbytehq/airbyte/pull/10186 But it is not tested. So, How can i test my changes. What I tried? In docker-compose.yml file there are multiple airbyte images used. So, I tried building the airbyte-bootloader image. It gave error

airbyte-bootloader.tar not found

. How can i make airbyte-bootloader.tar. How to make other dependencies(tar files) for airbyte images used in docker-compose.yml.

Titas Skrebė

02/09/2022, 3:23 PM

Hi, is it the expected behavior for a source, to ingest all data into a single column

_airbyte_data

instead of putting into the respective column names? I know that it expands into columns when I enable Basic Normalization, but I would like to know if it doesn't expand by default without normalization.

Jay Bujala

02/09/2022, 4:01 PM

Hi Airbyte team, I am having a problem trying to use gradle to build, really anything from a connector on up where the code-generator part fails. Python 3.9.10 and 3.7.x both throw the same message. All the other dependencies are installed, just can't get the code generator to build: Of note, I am on Win 10 terminal direct, not wsl Docker will build connector images ok Thanks in advance for any ideas or help!

Copy code

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':tools:code-generator:airbyteDocker'.
> A problem occurred starting process 'command 'C:\git\airbyte\tools\bin\build_image.sh''

* Try:
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':tools:code-generator:airbyteDocker'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:145)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:143)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:131)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:402)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:389)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:382)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:368)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'C:\git\airbyte\tools\bin\build_image.sh''
        at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
        at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
        at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
        at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'C:\git\airbyte\tools\bin\build_image.sh'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WindowsProcessLauncher.start(WindowsProcessLauncher.java:22)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
        ... 4 more
Caused by: java.io.IOException: Cannot run program "C:\git\airbyte\tools\bin\build_image.sh" (in directory "C:\git\airbyte\tools\code-generator"): CreateProcess error=193, %1 is not a valid Win32 application
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 8 more
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
        ... 9 more


* Get more help at <https://help.gradle.org|https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See <https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings|https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 17s
22 actionable tasks: 11 executed, 4 from cache, 7 up-to-date
PS C:\git\airbyte>

Will Sargent

02/09/2022, 8:23 PM

Hi, I have a question to the dependency between 2 API Streams in a source. I am using the Faros TypeScript CDK. I want to use email activities (schema link) and campaigns (schema link) as an example. An email activity is a child under a campaign (like users to teams). In the source stream definition, campaign_id is required to get the children activities. The final data schema contains everything under email activities PLUS campaign.type. My question is, the source connector definitely fetch campaigns, in order to get email activities. My question is, how can I reference another stream when the destination connector processes a stream? Or we should always treat the each writeStream independent?

Yiyang (Heap.io)

02/09/2022, 10:12 PM

Hi Team, I am building a custom api connector and running into an error for only one endpoint. I get this error message

Copy code

{"type": "LOG", "log": {"level": "ERROR", "message": "Encountered an exception while reading stream SourceRippling\nTraceback (most recent call last):\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 108, in read\n    internal_config=internal_config,\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 141, in _read_stream\n    for record in record_iterator:\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 213, in _read_full_refresh\n    for record in records:\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 356, in read_records\n    next_page_token = self.next_page_token(response)\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/source_rippling/streams.py\", line 23, in next_page_token\n    if bool(decoded_response.get(\"has_more\", \"False\")) and decoded_response.get(\"data\", []):\nAttributeError: 'list' object has no attribute 'get'"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing SourceRippling"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceRippling runtimes:\n"}}
{"type": "LOG", "log": {"level": "FATAL", "message": "'list' object has no attribute 'get'\nTraceback (most recent call last):\n  File \"main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 116, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 107, in run\n    for message in generator:\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 112, in read\n    raise e\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 108, in read\n    internal_config=internal_config,\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 141, in _read_stream\n    for record in record_iterator:\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 213, in _read_full_refresh\n    for record in records:\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 356, in read_records\n    next_page_token = self.next_page_token(response)\n  File \"/Users/marissa/github/airbyte/airbyte-integrations/connectors/source-rippling/source_rippling/streams.py\", line 23, in next_page_token\n    if bool(decoded_response.get(\"has_more\", \"False\")) and decoded_response.get(\"data\", []):\nAttributeError: 'list' object has no attribute 'get'"}}

and fatal message that “list” object has no attribute “get”, but this appears right after all the records are read and yielded as output. Another endpoint I made a stream for worked fine without having this fatal message, and the only real difference I could see between the two outputs was that this one was in dictionary format enclosed in curly brackets {}, while the one that was erroring out had a response in list format with [] brackets. To fix this, I tried adding an if case to the parse_response method in the base stream that would yield each row in a for loop if the response type was a list. This gave the same result with all the records yielded, and still getting the fatal list object has not attribute message at the end. Is this a known issue by any chance, or has anyone encountered something similar?

Marissa Pagador

02/10/2022, 5:04 AM

Hello team ! Is there a way to specify the order of the properties in spec.json?. Properties appear randomly in UI. I want it to appear in certain order like mandatory fields first, followed by optional.

Narender Kumar

02/10/2022, 5:18 PM

We are currently developing a REST source (iLevel) that uses OAuth2.0. I have looked through other examples suggested here on Slack, but it seems that all the examples uses a

refresh_token

, and that this is required if I want to use the

OAuth2Authenticator

class in the CDK. My source doesn’t have a refresh token. In my

check()

function, I can do a custom POST request and authenticate no problem. Is there an option to either: 1. Use the OAuth2Authenticator class without a refresh-token somehow. 2. Just do a custom Post in my streams function instead of using the OAuth2Authenticator (like I do in

check()

) 3. Is there just an entirely different way to do this 4. …or am I out of luck and Airbyte simply doesn’t support Oauth2 without refresh tokens?

Toma Bernotaitė

02/10/2022, 8:13 PM

Hey guys! I've just started working with Airbyte and currently I'm trying to extend existing Shopify connector in order to extract Customer Journey data via GraphQL. Does anyone have experience on implementing GraphQL streams on Airbyte and maybe have some examples to share? I'm fairly new to Python as well as Airbyte so it's a bit challenging. Thank you in advance 🙌

Toma Bernotaitė

02/10/2022, 8:51 PM

I don't see any information on connecting to SAP Sql Anywhere DB. This is not a database type that I normally connect to so I am not as familiar with it. Does airbyte have any connector that can already connect to an SAP SQL Anywhere DB as a source?

Zak Keener

02/10/2022, 9:58 PM

Hey all — posting this here to get some exposure: https://github.com/airbytehq/airbyte/issues/10252 Latest Plaid source supports syncing transactions, which is great, but it only syncs the last 100, whereas it should paginate them in order to sync all historical transactions. Details are in the issue. Would love some help developing this if possible!

Zak Keener

02/11/2022, 2:01 AM

Hello dear Airbyte team, I'm following the Python CDK Speedrun: Creating a Source and encountered an issue while testing the Check for sample_files/config.json. Below is the detail. Where the issue started (screenshot #1)

python main.py check --config sample_files/config.json

The error message (screenshot #2) The json context in
sample_files/config.json
(screenshot #3) What interesting is if I put the same json context (e.g. "pokemon_name": "pikachu") into

secrets/config.json

(I know it's not where it belongs but just for a testing), and the Check testing will run successfully (screenshot #4). Not sure if it has anything to do with Anaconda, but other than that I'm running out of idea of the root causes. Any support will be much appreciated!! Thanks in advance and let me know if any more info is needed.🙇‍♂️

will

02/11/2022, 10:29 AM

Hello Airbyte team, I want to trigger when reading data error and send the message stack to the crews via slack. Another, I want to note and retry only the failed stream_slice (not at all). I'm new to Airbyte so it's a bit challenging. Thank you in advance 🙌

Hưng Lê

02/11/2022, 3:04 PM

I am getting so close to finishing my first connector. 🥳 Thanks to the excellent support on this community! One of my streams has heavy limitations from the source system. The current dataset have >2mill rows, and the system only allows maximum 100 pages, with 1.000 records each - per query Not a huge issue, as I have been able to segment the data, and by manipulating

def next_page_token

I am able to build a list of different filter params that are well under the limits. But, here is my issue: • As soon as I run my

main.py read

script, it kicks off the request on the endpoint. (this query doesn’t work, as it tries to return all the 2 million rows). • I need this stream to wait for the next_page_token to make a few secondary requests - to build a list of filters - before starting to pull from the dataset. What is the best way to do this?

Greg Blackman

02/11/2022, 6:19 PM

Hello! It's possible to configure de datetime format like YYY-mm-dd to dd-mm-YYY on the Json Schema. Something like this:

"createdAt": { "type": ["null", "string"], "format": "date-time"  "format_string": "dd-mm-YYY HH:MM"}

Antonio Grass

02/13/2022, 1:10 AM

Hello everybody Im getting

ERROR: (gcloud.beta.compute.ssh) Could not fetch resource:

all the time

Enrico Tuvera Jr.

02/14/2022, 4:59 AM

wasn't there a zoho source connector at some point? or am i remembering things wrong?

Enrico Tuvera Jr.

02/14/2022, 11:54 AM

Hi Team, I am trying to locally build airbyte. Receiving this error:

In the thread.

Tomas Balciunas

02/14/2022, 5:19 PM

Airbyte team, is there any schedule on when Twitter (social and ads) and Linkedin (social) connectors will be released? Some conversations on your GitHub suggest that at least Twitter is completed

Tomas Balciunas

02/14/2022, 8:51 PM

I have my connector built but run into trouble generating the docker file.

pip install --prefix=/install .

gives an error:

error: could not create '/install': Read-only file system

. Seems like the error is happening because it's trying to install at the root file level

macos won't let you install anything there but it was generated like this so I think I'm missing something.