Hi @Jon Tan (Airbyte), having an issue with PbA oauths for our initial connector tests:

Slack

Notion

GitHub

Intercom

Hubspot

Example below. Mind helping resolve? The support email we got back asked to send them all the redirect URLs, but I imagine there is some setting or account issue. We hope to have working oauths for the connectors we use with Airbyte without needing to shoot your support the redirect URLs each time 🙏. Recommended to reach out by the BD / product teams. Account: benton@brillai.co.

Yes, we’ll work on getting this set up for you. From your list: • Slack - in progress • Notion - should be supported • Github - unfortunately can’t be supported currently due to some technical limitations on our end • Intercom - in progress • Hubspot - should be supported For the ones on the list in-progress, I’ll post updates here as the team works through getting the correct redirect URLs added for cloud. For the ones that are already supported, please let me know if you run into issues using them.

Thanks @Jon Tan (Airbyte). Is there a way for us to know which Airbyte connectors don't offer oauth API access? In Fivetran, they mark API access with 'API Configurable'. For Airbyte, is it Airbyte Open Source? GitHub Airbyte Cloud has 'OAuth' listed https://docs.airbyte.com/integrations/sources/github/?__hstc=27854691.0bdb11dd191aa7e[…].&_ga=2.62583916.1155226986.1693859914-1004140627.1693859914

@Benton Turner As you’ve already figured out, a connector supporting OAuth doesn’t necessarily mean we’ll be able to currently support OAuth via the API in cloud today. To go into the technicals a bit, the OAuth application that gets created when we register Github only allows for a single callback URL. We haven’t yet done the work to create a single consolidated OAuth callback URL in our system for both the API and the UI so for now they’re separate. At the same time, our system doesn’t yet have a way to support setting up multiple OAuth parameters for a given context (workspace, connector) so for Github, we can only have one set of OAuth credentials available and in this case, it’s the one that holds the redirect URL for the UI. In order for us to be able to support Github OAuth via the API, we’d need to either consolidate our OAuth URLs or create some way to set up multiple OAuth parameters for workspace/connector and then be able to propagate enough context through to our jobs so that the job runners know which set of credentials to use. Both options are pretty high lift and given our other commitments on the teams that would work on this, it seems unlikely we’ll have a solution by EOY. In terms of what is possible today, any connector that allows multiple redirect URLs to be registered will allow OAuth via the API. We’ve been adding the API callback URL to those as they come in so any that you need, just request them in this channel and I’m happy to work on adding them for you (except Github). The only connectors I’m aware of that won’t be able to support OAuth without the investment I described earlier are as follows: • Airtable • Github • Harvest • Mailchimp • Smartsheets • Snapchat Marketing • Snowflake • Square All of these have other methods of authentication that are available through the Airbyte API (i.e. Github source can be authenticated via personal access tokens).

@Jon Tan (Airbyte) As far as providing a callback URL to you guys, can that be a single url for all connectors I.e.

<https://app.brillai.co>

?

@Nik Weinmeister Yes, so as long as the page hosted at

<https://app.brillai.co/airbyte>

is capable of both receiving a secret ID in the query string and tying that to the user session so you know which connector the secret belongs to, you could theoretically use the same page for all your redirects.

@Jon Tan (Airbyte) That sounds good, I was able to setup a successful oauth flow with Notion last night so can confirm that that works. So if we want to go ahead and get Slack working as our next priority, can we set our redirectURL as https://app.brillai.co/airbyte_oauth/:source_type so for slack specifically: https://app.brillai.co/airbyte_oauth/slack ? This page is setup to pull out the secret ID from the query string and will have user sessions stored in order to make the request to then create the actual source

@Nik Weinmeister Yes, you won’t need to register anything with us, when we have slack working with API OAuth, you’ll just be able to

initiateOAuth

same as you have been with your

<https://app.brillai.co/airbyte_oauth/slack>

URL as the passed in

redirectUrl

. As far as the Hubspot error, you should not need to pass in the credentials object as your secret should be expanded into the credentials object automatically. Let me take a closer look to see what’s going on there.

Thank you!

@Nik Weinmeister can you please send me the rough timestamp of when you created the source?

Looked in the db quickly to see the source and the configuration didn’t look right. Checked into the request and it appears you’re passing the

secret_id

in the

configuration

object in the body, but it should be in a completely separate key in the body. That inconsistency is something we’re hoping to iron out in the near future, but haven’t gotten to yet.

Ahh okay, let me try that with GA

I’m surprised that Notion is working with that in the configuration object. Do you have a specific connection that seems to be succeeding that I can take a look at?

Connection with Notion: ea316400-6994-435f-a2a7-4a20ec406c5d

I would not expect that source to work correctly unless you recreate it with the secret_id outside the configuration block

🤦 I probably should've called that out since all the other keys are camelcase

I myself forget all the time and I wrote the code so really I should have remembered that sooner. Let me know once you’ve created it and I can check the logs & db.

so adding secretId it fails on source creation now

Yes,

credentials.auth_type

needs to be set to

Client

per the connector specification. I do not see a

credentials.token

in the connector specification so I’m not really sure where that would be coming from. Can you try adding the

auth_type

of

Client

and try again?

in the docs credentials is inside the configuration object is that correct?

ohhh I’m seeing this is for notion, not google analytics, correct?

Correct I'm hitting notion right now

Your request body should end up looking like this:

{
  "configuration": {
    "sourceType": "notion",
    "start_date": "2020-01-01T00:00:00.000Z",
    "credentials": {
      "auth_type": "OAuth2.0"
    }
  },
  "name": "Test Organization - notion",
  "workspaceId": "ba0aa7d2-71fe-4488-8f78-651336f3155a",
  "secretId": "secret_id_here"
}

The

auth_type

of

Client

was operating under the assumption that we were still talking about google analytics.

gooot it, okay that appears to have worked

I’ll post some info here on how I go about building these requests to create OAuth sources. We can use Notion as a case study for you and any future folks who come searching through here. If we’re trying to create a Notion source, after running through the

initiateOAuth

endpoint and retrieving the secret, we want to create a source. I usually go to the open source Airbyte repo containing connectors and look for the source specification for the source I want to create. In this case https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-notion/source_notion/spec.json. The important points in the spec are any constants (in this case, we want to use the OAuth2.0 version of the credentials block so we need to add that

credentials.auth_type

into our source configuration when we create it). If we look further down in the

advanced_auth

key, any properties that appear in either the

advanced_auth.complete_oauth_output_specification

or the

advanced_auth.complete_oauth_server_output_specification

will be either provided by the secret you obtained via

initiateOAuth

OR will be injected by the platform/overriden at the workspace/organization level. So we can confirm that the only property that needs to be present in the

configuration.credentials

block when creating a Notion source via the Airbyte API is the

auth_type

key. tldr; any constants found in the credentials block need to be added in the

configuration

when you’re creating a source. I don’t know of any examples off the top of my head where you would need to add something that isn’t a constant, but also isn’t in one of the relevant

advanced_auth

sections.

@Jon Tan (Airbyte) This is super helpful, this definitely unblocks me on multiple sources, running through each of em now

Awesome! This is something that could definitely be more consistent and well documented so your feedback is appreciated. I’ll pass some feedback to the appropriate team and will add some tickets to our own backlog to see if we can smooth over some of this to improve the user experience. Between that and some documentation hopefully this gets a little better.

no problem! definitely understand the early stage inconsistencies. I guess my only followup on all this is just is there plans on sort of making some of the credentials keys/values consistent across all the sources?

I think we hope to make that more consistent at the specification level but that’s a different team and I’m not currently sure of their timeline although they’re very busy right now so my guess is that it won’t be this year. Having said that, we (the team behind the Airbyte API) can take a stab at adjusting the API interfaces to make these things easier. In this case I expect we’d just add the constants to the credentials blocks automatically in the API server so that users don’t need to add them manually (since they already don’t need to add anything else) so it hopefully won’t really be a breaking change. For breaking changes to the API contract, we do try to communicate those ahead of time although we’ve had relatively few so far and haven’t had to establish formal process. Tagging @Riley Brook (Airbyte) here in case he has follow-up thoughts/information on that process. Important to note that if we do introduce a breaking change to the API we won’t be changing the underlying data structures that back your sources/destinations/connections so those should not break.

Awesome yea that's perfect for now, not a huge deal on our side to have the dynamic creds for now.

Ah ok - no worries, we are literally here to receive bombardment! If you have any other questions don’t hesitate to reach out!

@Jon Tan (Airbyte) Thanks for your help and support! We had a quick chat with Riley and would love to get a sense of ETA for the following Oauth connections: • Slack - TBD • Notion - READY • Github - 2024? • Intercom - TBD • Hubspot - READY • Posthog - TBD • Sentry - TBD • Asana - TBD • Activecampaign - TBD • Monday.com - TBD • Google Analytics - READY Then more broadly speaking, would be great to know when you might add the following API based connectors (whether API key or Oauth, but ideally Oauth): • CircleCI • Figma • Google Drive • Fireflies • Loom • Discord • Hyros The above aren’t supported by Fivetran or Airbyte, so would be a competitive edge if you built them out.

Hey @Benton Turner still working on an ETA for the OAuth connectors we can support as we’ve had some personnel outages on our side. For the connectors that we don’t already have, the timing of that is based on either community desire to develop a public connector or prioritization of a different team. I’ll reach out to that team and see if any of these have already made an appearance on their radar/backlog. @Riley Brook (Airbyte) feel free to fill in anything I’ve missed.

Great, thanks @Jon Tan (Airbyte). Let us know when you have a chance regarding the ETA of the above

Just touched base with the team and it sounds like we should be able to get those resolved by EOW.

That's great news, thanks @Jon Tan (Airbyte). Excited for it, let us know if we can help in any way.

Hey @Benton Turner Intercom & Posthog have been updated. Still waiting on some of the others, we may have status updates on some of them today. Unsure of whether or not there will need to be a revised ETA but will let you know when I know more.

• Thanks @Jon Tan (Airbyte), listing the rest for convenience here: ◦ Slack - TBD ◦ Notion - READY ◦ Github - 2024? ◦ I~ntercom - TBD~ ◦ Hubspot - READY ◦ Posthog - TBD ◦ Sentry - TBD ◦ Asana - TBD ◦ Activecampaign - TBD ◦ Monday.com - TBD

Sentry and Asana are complete, I just pinged about Slack and Monday which were in-progress.

@Jon Tan (Airbyte) Just tested these out Slack: still getting same error, looks like user_scope is still empty in the redirect URL:

user_scope=&

Sentry: Getting the generic 500 from the initiate Oauth API request (ex: timestamp Oct. 12, 2023 221732.047 ) Asana: Got an error from the redirect URL in Asana: "invalid_request: The

redirect_uri

parameter does not match a valid url for the application."