Hi team im curious about the usage of connectors and connections via the Airbyte REST API idea : we would like to create a connection as kind of a "template" beforhand (either manually via the gui or via the airbyte api). say source is a rest api from a service and destination are tables in postgres. then we would like to trigger a sync with this connection, but to be able to "override" the config for this this api call (just to be able to call the connection with other arguments) currently, i don't see in the api docs, that this is possible. we would start the sync of the connection via dagster => and there also dagster has only parameters to set the host and post of the airbyte instance and the id of the connection, but no parameters specific for the source. is this somehow possible via api ? because when we test a connector written in python, we then also pass in the config via the command line e.g. "python myconnection.py check --config someconfig.json" => would it be possible to adapt this to a connection via rest api call ?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2934 to ensure timely resolution!

Hey @Patrik Deke, so I’m understanding this correctly. You want a way to dynamically change the config of a particular source? Have you looked into this endpoint: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/sources/update?

Hey @Saj Dider (Airbyte) Thanks for reaching out ! It's kind of what i want. i don't want to beforhand update the connection, to be able to do then the sync. i would like to start the sync with specific configurations/arguments for just this one sync. say i have a connection, which can get address inforamation of a person by her social security number (ssn) and then saves them into a destination (e.g. postgres) i would like then to be able to trigger a sync for this connection via rest api and give the ssn as a parameter.

Yes, I think I understand. I don’t believe there is a way to do this elegantly with our API. Airbyte is best suited for bulk extraction and loading. The alternative you suggested is what I also thought of but it’d be terribly inefficient. It is worth a try though if you are only extracting a small number of records. My other guess would be that you can modify the connector to take in a list of objects as part of the config or a file that contains all of the ssn data and then modify the methods for the extraction part to read inputs from the file. This solution doesn’t sound very elegant either.

unlucky to here, but i tought so because of api doc but also glad, that we have the same understanding 👍 do you think, that this funcionality is interesting for you (airbyte and/or community) so that it could be implemented in the future ? because during development of the connector, we also give the configuration via the cli e.g. "python main.py check --configuration someconfig.json" => would it not be kind of "simple" to implent this behaviour via the rest api (i know, nothing is simple, but theoretically ;D)

Thanks @Patrik Deke, I can forward this request to our product team, who may have a better understanding of whether this is feasible. Ultimately, we prioritize features based on impact and community need. If this is a highly requested feature, we will heavily consider it. I understand what you're getting at with the creation of a layer on top of the source connector, but I'm unsure if this is something Airbyte is suited for and if there are more specific tools that can achieve what you're looking for. Our API is still very much in beta so of course, the scope of the API and its utility is subject to change based on our community's problems. I encourage you to raise issues in our Github for features you'd like to see for the API.

Hi @Saj Dider (Airbyte) thank you for your answer we need to look into it with our team but i will look out to do a more detailed description to create an issue/feature request on github thanks for your help !