Hello, I’m using octavia with development (macOs) and production (digital ocean) environments. My workspaces have different UUIDs, as well as all the corresponding sources/destinations/connectors, their UUIDs dont match between dev and prod. This surfaces two issues: 1. the workspace state files (in say,

sources/state_uuid_here.yaml

) dont match so it creates duplicate connectors (one for each statefile it seems?). My current workaround is I added the

state_my_dev_uuid.yaml

and

state_digital_ocean_workspace_uuid.yaml

to the

.gitignore

so each can have its own state without polluting the other, this seems to be working. 2. I’m also using Dagster (with the dagster-airbyte package), and to initialize the airbyte assets in dagster, I’m using the UUIDs of sources, destinations, connections. But since these don’t match between dev/prod, I’ll need to add a toggle based on the environment for which to use, which is just added overhead. Otherwise, calling airbyte in prod fails since its trying to hit up UUIDs for the dev connectors which dont exist in prod. Any suggestions on this workflow, cause its becoming increasingly time consuming to manage. I’m using

octavia apply --force

for example in both environments, hence posting here.

Hi Dayten - We wrote a deploy script as a wrapper around Octavia because it doesn't work well out of the box. A few things it does (or will do soon) that overlap with what you are wrestling with: • It will parameterize configuration files based on local/dev/prod deployment. For example, it will change what databases it connects to so you only have one "my-database-dest" but it will be wired up differently in each env. It does this via standard environemnt variables, just using them in a different manner than the traditional use for secrets. • It will push and pull state files from S3 so that they don't need to be checked into git and shared across deployment environments. • I've not used the dagster tools, so I don't know where exactly they get tripped up by UUIDs. But one thing we've found is that our custom connectors have different UUIDs, so we query by name to extract out the UUID in each env and parameterize that in our configurations. • It pulls secrets from AWS SSM that are parameterized by environment. We've been able to cobble this together in a working state, but Octavia feels like a dead end at t his point. I've filed Github issues, asked on web forums, asked here on slack, talked to folks at Airbyte, and there is no response or progress on issues that have been open for many months, let alone make improvements to make it enterprise-ready. It sounds like there is an intent to move to a terraform provider (which would be great), so my recommendation is to get a workflow that works for you (using Octavia and/or dagster) and wait to see how that effort developers.

Hey @Dayten Sheffar did you ever find a workable solution? I'm facing a similar issue - I want to have 2 environments (dev & production) on 2 separate git branches, pointed at 2 different EC2's, but am running into the limitation with state files as well • the connection state file is overwritten when using

octavia apply --force

- but I need different state files for each environment (i.e.

dev_state_uuid.yml

and

prd_state_uuid.yml

). If I added the state files to the

.gitignore

, duplicate connections are created every time I ran

octavia apply --force

via github actions.. Are you running

octavia apply --force

locally?