Hello!octavia wave I've been trying out the Terraform provider to setup sources, dests and connections when upgrading Airbyte. Works fine! Is it possible to export existing resources into Terraform resources somehow? Another thing that would be nice would be to keep the stream states in Terraform, that way you could apply the most recent state when migrating and avoid having to start off with a full refresh. Right now I'm using a script that reads state from the old instance and updates the state of the new instance, using the API.

The import feature is an often requested feature that is not available yet unfortunately

Thanks! As I said we got a working solution for the second request, just would be nice if we could utilize terraform for the state as well.

Hey @Carl Raunås, Right now I'm using a script that reads state from the old instance and updates the state of the new instance, using the API. would you mind sharing how you handle this with the Airbyte API? which endpoints are you using? thx ! 🙏

Sorry for the late reply! Did you already find a solution for this, otherwise I could send something during the day!

Hey @Carl Raunås, No worries at all and thanks for coming back to me! No I haven't been able to find something really worthy. That would be amazing, thank you 🙏 !!

Ok, so this might be a bit specific to fit our setup. Assuming it's running on localhost I get the workspace id on this endpoint:

<http://localhost:8000/api/v1/workspaces/list>

Getting the connections on workspace

<http://localhost:8000/api/v1/connections/list>

with payload

"workspaceId": workspace_id

Getting the state of each connection

<http://localhost:8000/api/v1/state/get>

with payload "`connectionId": connection_id` Im saving all states together with the name of the connection to a textfile to be used for setting the state on the other instance. The new instance will already have all source. destinations and connections setup, preferably using Octavia or Terraform. Get workspace id of the workspace on new instance. Get all connection ids on the new instance. Read the saved states from file. The saved state file contains the connection id of the old instance, so here I'm replacing the connection id of the state by matching connection state on connection name. Something like this:

[conn["connectionId"] for conn in connections if conn["name"] == state['connectionName']]

and then replacing this connection id of the saved state object. For each connection on new instance update the state

<http://localhost:8000/api/v1/state/create_or_update>

with payload

"connectionId": connection_id,

"connectionState": state

Think this is the docs I used to do this: https://reference.airbyte.com/reference/listworkspaces

It's amazing, thanks for the detailed answer @Carl Raunås. no it's all super clear !

Great! Happy to help!