Hey folks; has anyone successfully used their build tool to run

octavia apply

? I’m setting up a deployment pipeline using CircleCI and just wondering what the best approach for managing those state files is. I could use CircleCI’s caching service for example but then that would prevent devs being able to run octavia commands locally if there is a need for manual intervention. I assume there’s no S3 or GCS backend yet like terraform has for it’s state file management?

Hi team, I just wanted to know if this issue is intended to be solved or the feature is not in roadmap, I could work on it.

Hello guys, Have you encounter this problem when trying to generate destination yaml,recently

TypeError: _from_openapi_data() missing 2 required positional arguments: 'supports_dbt' and 'normalization_config'

the command octavia --airbyte-url http://localhost:8085 generate destination 471e5cab-8ed1-49f3-ba11-79c687784737 weather

Hey folks! Just submitted a super-dummy issue (https://github.com/airbytehq/airbyte/issues/22567), and I wanted to collaborate. As per the docs I could assign to my self the ticket, but I think I should be a collaborator or something, shouldn't I? Thanks !

Hey team - can Octavia be used to managed the connector versions at all? From local testing it appears as if the connector version in the octavia configuration yamls is functionally redundant, i.e. if I change it there’s no change performed on the instance.

Is it possible to upgrade airbyte using Octavia to store the configuration ? If not, is there any plan to ?

I'm unable to pass in environment variables for passwords... My

~/.octavia

file looks a bit like this:

AIRBYTE_URL=<http://localhost:8000>
DEV_PASSWORD=12345_not_real

And in the

source

file I have

resource_name: test_apply_deakin
definition_type: source
definition_id: b39a7370-74c3-45a6-ac3a-380d48520a83
definition_image: airbyte/source-oracle
definition_version: 0.3.22
configuration:
  host: <http://ora-test-admit-b.c6afqudcyert.ap-southeast-2.rds.amazonaws.com|ora-test-admit-b.c6afqudcyert.ap-southeast-2.rds.amazonaws.com>
  port: 1521
  schemas:
  - DEAKIN
  password: '{$DEV_PASSWORD}'
  username: blah
  encryption:
    encryption_method: unencrypted
  tunnel_method:
    tunnel_method: NO_TUNNEL
  connection_data:
    service_name: db1
    connection_type: service_name

I am using the very latest of both octavia/airbyte (version 0.40.32), and Oracle source (0.3.22) For some reason when running

octavia apply

it is not updating the password. I have tried it with/without single quotes and with no quotes crashes In an odd twist, if I manually put the password in the source file directly, it updates fine, and the source validates/passes connection test - so its definitely an issue with pulling it in as an environment variable. Any ideas?

Hi team ✋, I am trying to generate Snowflake destination in Octavial CLI (version 0.40.32) using the following command

octavia generate destination 24892c4-daac-4491-b35d-c6688ba547ba snowflake

but errored out with the below message. I am using Airbyte 0.40.32. Could you please help me to get it resolved? Thanks in advance.

🐙 - Octavia is targetting your Airbyte instance running at <http://localhost:8000> on workspace 6f1bedf4-795e-450b-910e-45eb306ecf56.
Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/commands.py", line 42, in destination
    generate_source_or_destination("destination", ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], definition_id, resource_name)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/commands.py", line 22, in generate_source_or_destination
    definition = definitions.factory(definition_type, api_client, workspace_id, definition_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/definitions.py", line 148, in factory
    definition = DestinationDefinition(api_client, definition_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/definitions.py", line 64, in __init__
    self._api_data = self._read()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/definitions.py", line 72, in _read
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/definitions.py", line 68, in _read
    return self._get_fn(self.api_instance, **self._get_fn_kwargs, **self.COMMON_GET_FUNCTION_KWARGS)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api/destination_definition_api.py", line 782, in get_destination_definition
    return self.get_destination_definition_endpoint.call_with_http_info(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 849, in call_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 410, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 204, in __call_api
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 197, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 456, in request
    return <http://self.rest_client.POST|self.rest_client.POST>(url,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/rest.py", line 269, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/rest.py", line 228, in request
    raise ApiException(http_resp=r)
airbyte_api_client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.23.3', 'Date': 'Mon, 20 Feb 2023 15:47:23 GMT', 'Content-Type': 'application/json', 'Content-Length': '296', 'Connection': 'keep-alive'})
HTTP response body: {"message":"Bad Request","_links":{"self":{"href":"/api/v1/destination_definitions/get","templated":false}},"_embedded":{"errors":[{"message":"Required argument [DestinationDefinitionIdRequestBody destinationDefinitionIdRequestBody] not specified","path":"/destinationDefinitionIdRequestBody"}]}}

good morning !, im trying to appy all, the connections, sources, and destinations, when I run octavia apply, the sources and destinations works, but te connection between them fails:

irbyte_api_client.exceptions.ApiTypeError: Invalid type for variable 'non_breaking_changes_preference'. Required value type is NonBreakingChangesPreference and passed type was str at ['non_breaking_changes_preference']

I'm using octavia-cli 0.40.32 and Aribyte 0.40.32

When is Octavia expected to go to beta or ga?

hey guys, any one uses octavia-cli like a pod running on kubernetes ?

Which should be the best way to use octavia-cli into EKS environment ?

Hello, I had connections that worked fine via input in the UI. I installed octavia 0.41.0 (same as my airbyte version), ran the

init

and

import all

functions, set the relevant secrets in my

~/.octavia

file, I entered the secrets in the relevant

configuration.yaml

files, and I know this is working since for many non-secrets it is applying (and updating) these in the UI (i.e., if I set some non-secret env file to ABC123, and update it to ABC1234 in

.octavia

, it changes in the UI after running

apply --force

). However, when I test the connections, it is failing, so it must be on the private key. One example in the .octavia file is for the AppStore .p8 file

APPLE_P8="-----BEGIN PRIVATE KEY-----\nSOME\nSTUFF\nHERE\n-----END PRIVATE KEY-----"

I’m out of ideas, solutions that have worked for various people (posted here on Slack) are not working for me. Relevant logs for failing connection test:

2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 116, in post
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 -   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 319, in connect
2023-03-07 19:22:22 ERROR i.a.c.i.LineGobbler(voidCall):114 - ImportError: sys.meta_path is None, Python is likely shutting down
2023-03-07 19:22:23 WARN i.a.w.g.DefaultCheckConnectionWorker(run):107 - Check connection job subprocess finished with exit code 1
2023-03-07 19:22:23 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):124 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Error checking connection status: no status nor failure reason were outputted
	at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:198) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:117) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:41) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.41.0.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Hi Airbyte Community. I have recently started working on Airbyte (replacing Singer API). Issue i am facing is that while trying to sync data from mongo db to AWS s3 location, few columns are being missed in tables. For instance, Consider a table users with columns user_id, name, location,phone_num While the sync is performed, column 'location' is missed, while it is present in mongo db. I tried to figure out this issue via Octavia CLI, but could not find any way that mising columns can be defined. Just wanted to mention that this column has ~70% nan values. But going with the fact that even for 30% data the column should not be missed. It would be very kind of you guys if anyone can show some way out of this. cc: @Sachin Tripathi @Vaidehi Khalpada

I just saw that https://github.com/airbytehq/airbyte/pull/22654 got completed this morning. Has the package been automatically pushed up to PyPi? Not sure if my search skills are failing me on there, or if it's still so fresh that it isn't up there yet. 🙂

I am facing the issue (https://github.com/airbytehq/airbyte/issues/22478): Am able to generate source but unable to generate the destination due to error - TypeError: _from_openapi_data() missing 2 required positional arguments: 'supports_dbt' and 'normalization_config' Any help would be appreciated

Hi I have created some custom sources and added them through the Airbyte UI

settings >> Sources >> New connector

Can I include them via Octavia CLI? or any other way without using UI

Hello.. I am new bee to airbyte.. trying to install octavia-cli.. using both direct install.sh and docker run.. after execution of docker-run when I type octavia I am getting command not found.. I could not see docker running in docker ps.. can you please help

morning guys, anyone knows why octavia-cli 0.40.29 with airbyte 0.40.29 when I run "import all" I get "aborted" ⚠️ - Please update any secrets stored in sources/ips_stg/configuration.yaml ✅ - Imported source IoT Klika Staging in sources/iot_klika_staging/configuration.yaml. State stored in sources/iot_klika_staging/state_b8f15981-7237-49ab-a2a8-072ee5835262.yaml ⚠️ - Please update any secrets stored in sources/iot_klika_staging/configuration.yaml ✅ - Imported source IoT Klika Prod in sources/iot_klika_prod/configuration.yaml. State stored in sources/iot_klika_prod/state_b8f15981-7237-49ab-a2a8-072ee5835262.yaml ⚠️ - Please update any secrets stored in sources/iot_klika_prod/configuration.yaml Aborted!

it looks like recent versions of octavia have a mismatch in their docker tags. any idea why? the most recent image available is 0.42.1, yet master has 0.42.0

Hi all, I've been encountering some issues with Octavia and have tried asking on discuss.airbyte.io and/or filing Github issues without any response (one of which has been open for more than 5 months). Hoping someone here might be able to help: 1.

octavia import connection foo

fails Github issue: https://github.com/airbytehq/airbyte/issues/23953 Airbyte discussion topic: https://discuss.airbyte.io/t/octavia-get-import-connection-is-completely-broken/4199 Octavia will fail outright if you try to import/get a connection by name (even if it doesn't exist). Importing by UUID works, which is the workaround I've put in place. 2.

octavia apply

can't send certain fields that

octavia import

creates Github issue: Airbyte discussion topic: https://discuss.airbyte.io/t/unable-to-apply-airbyte-configuration-using-octavia-cli/4138/2 Github issue: https://github.com/airbytehq/airbyte/issues/19768 There are three fields (

geography

,

notify_schema_changes

, and

non_breaking_changes_preference

) that are included when you do an import, but then fail if you try to apply them back. You can manually remove these fields and rely on the default values, but then apply will think there are changes to be applied every single time.

Oh, and this one as well: https://discuss.airbyte.io/t/unable-to-specify-snowflake-private-key-as-secret-through-octavia-cli/4139 Because Octavia relies on docker environment files for passing information into the container, it seems impossible to pass multi-line secrets, most notably Snowflake private keys. Is there any way around this short of creating a custom container with a different

ENTRYPOINT

? It would be nice if you could pass the secrets configuration via a more robust method (such as a YAML/JSON file).

Hey guys! Check out my post on how with the use of Octavia CLI and a piece of Python code to set up multiple sources programmatically in case you have a multitenant database architecture. Thanks to @Bob De Schutter for the idea. Let me know what you think. Feel free to connect!

Has anyone encountered this issue?

airbyte_api_client.exceptions.ApiAttributeError: AirbyteStreamConfiguration has no attribute 'suggested' at ['received_data']['catalog']['streams'][0]['config']['suggested']

As far as I know, I'm using compatible versions of airbyte and octavia. I'm on airbyte 0.42.0.

Hi 👋 I was looking at Octavia CLI's docker compose configuration and noticed that it mentions

airbyte/octavia-cli:latest

for image name but in CLI Registry,

latest

tag doesn't exist and thus, running

./run-ab-platform.sh

fails once compose file is updated to include CLI service, I had manually use latest tag version

0.42.1

to make it work. 😅

Hi all, How can I use octavia-cli to refresh schemas of an already configured connection? Are there any best practices around this use case?

Hi all! I noticed no new octavia cli releases have been pushed to the docker registry. The last version is 0.42.1. Is this a known issue? Is this the right place to report?

Hello, I’m using octavia with development (macOs) and production (digital ocean) environments. My workspaces have different UUIDs, as well as all the corresponding sources/destinations/connectors, their UUIDs dont match between dev and prod. This surfaces two issues: 1. the workspace state files (in say,

sources/state_uuid_here.yaml

) dont match so it creates duplicate connectors (one for each statefile it seems?). My current workaround is I added the

state_my_dev_uuid.yaml

and

state_digital_ocean_workspace_uuid.yaml

to the

.gitignore

so each can have its own state without polluting the other, this seems to be working. 2. I’m also using Dagster (with the dagster-airbyte package), and to initialize the airbyte assets in dagster, I’m using the UUIDs of sources, destinations, connections. But since these don’t match between dev/prod, I’ll need to add a toggle based on the environment for which to use, which is just added overhead. Otherwise, calling airbyte in prod fails since its trying to hit up UUIDs for the dev connectors which dont exist in prod. Any suggestions on this workflow, cause its becoming increasingly time consuming to manage. I’m using

octavia apply --force

for example in both environments, hence posting here.

Hi guys! Will we have a python wrapper for the octavia-cli?

on

octavia apply

File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/rest.py", line 226, in request
    raise ServiceException(http_resp=r)
airbyte_api_client.exceptions.ServiceException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.23.3', 'Date': 'Tue, 18 Apr 2023 21:41:28 GMT', 'Content-Type': 'application/json', 'Content-Length': '13965', 'Connection': 'keep-alive'})
HTTP response body: {"message":"Internal Server Error: SQL [select (exists (select \"public\".\"connection\".\"id\", \"public\".\"connection\".\"namespace_definition\", \"public\".\"connection\".\"namespace_format\", \"public\".\"connection\".\"prefix\", \"public\".\"connection\".\"source_id\", \"public\".\"connection\".\"destination_id\", \"public\".\"connection\".\"name\", \"public\".\"connection\".\"catalog\", \"public\".\"connection\".\"status\", \"public\".\"connection\".\"schedule\", \"public\".\"connection\".\"manual\", \"public\".\"connection\".\"resource_requirements\", \"public\".\"connection\".\"created_at\", \"public\".\"connection\".\"updated_at\", \"public\".\"connection\".\"source_catalog_id\", \"public\".\"connection\".\"schedule_type\", \"public\".\"connection\".\"schedule_data\", \"public\".\"connection\".\"geography\", \"public\".\"connection\".\"notify_schema_changes\", \"public\".\"connection\".\"non_breaking_change_preference\", \"public\".\"connection\".\"breaking_change\", \"public\".\"connection\".\"unsupported_protocol_version\", \"public\".\"connection\".\"field_selection_data\" from \"public\".\"connection\" where \"public\".\"connection\".\"id\" = cast(? as uuid)))]; ERROR: column connection.notify_schema_changes does not exist\n  Position: 683","exceptionClassName":"org.jooq.exception.DataAccessException","exceptionStack":["org.jooq.exception.DataAccessException: SQL [select (exists (select \"public\".\"connection\".\"id\", \"public\".\"connection\".\"namespace_definition\", \"public\".\"connection\".\"namespace_format\", \"public\".\"connection\".\"prefix\", \"public\".\"connection\".\"source_id\", \"public\".\"connection\".\"destination_id\", \"public\".\"connection\".\"name\", \"public\".\"connection\".\"catalog\", \"public\".\"connection\".\"status\", \"public\".\"connection\".\"schedule\", \"public\".\"connection\".\"manual\", \"public\".\"connection\".\"resource_requirements\", \"public\".\"connection\".\"created_at\", \"public\".\"connection\".\"updated_at\", \"public\".\"connection\".\"source_catalog_id\", \"public\".\"connection\".\"schedule_type\", \"public\".\"connection\".\"schedule_data\", \"public\".\"connection\".\"geography\", \"public\".\"connection\".\"notify_schema_changes\", \"public\".\"connection\".\"non_breaking_change_preference\", \"public\".\"connection\".\"breaking_change\", \"public\".\"connection\".\"unsupported_protocol_version\", \"public\".\"connection\".\"field_selection_data\" from \"public\".\"connection\" where \"public\".\"connection\".\"id\" = cast(? as uuid)))];

Octavia/Airbyte versions tested 0.40.32 / 0.41.0 0.41.0 / 0.41.0 0.41.0 / 0.42.1 0.42.1 / 0.42.1 0.42.1 / 0.44.0 the breaking field:

notify_schema_changes

is present in my config file, but fails when commented out as well. Also broken in the UI when submitting a new connection (local postgres to big query)