Airbyte #help-api-cli-orchestration

Ignacio Valdelvira

07/22/2022, 10:28 AM

Hi all, I’m playing around with octavia cli and created a postgres source with username/password/database set up as environment variables like this

Copy code

database: ${POSTGRES_PHLO_DATABASE}
password: ${POSTGRES_PHLO_PASSWORD} 
username: ${POSTGRES_PHLO_USERNAME}

and defined in my .env file. I assumed I’ll need to map those variables in the airbyte docker compose yaml too like this

Copy code

environment:
      - POSTGRES_PHLO_DATABASE=${POSTGRES_PHLO_DATABASE}
      ...

I wasn’t sure in which service I should do the mapping so I put it in all of them just to try However, once in Airbyte UI I can’t seem to connect to my postgres source. If I replace the database/password/username by hardcoded real values it works, so the problem seems to be that I can’t figure out how to correctly map the environment variables with the actual values. How can I fix this? Thanks!

Abba

08/09/2022, 8:25 AM

Can we edit connection states from octavia?

Piotr Sierkin

08/12/2022, 12:47 PM

Hello 😄 I’m doing research on how to secure airbyte and use octavia cli I found we can use gcp secret menager to store secrets reference here:

Copy code

<https://docs.airbyte.com/operator-guides/configuring-airbyte/#secrets>
SECRET_STORE_GCP_PROJECT_ID - Defines the GCP Project to store secrets in. Alpha support.
SECRET_STORE_GCP_CREDENTIALS - Define the JSON credentials used to read/write Airbyte Configuration to Google Secret Manager. These credentials must have Secret Manager Read/Write access. Alpha support.

And here is my question is possible to use octavia with this secret menager: The possible workflow I create the source where I put env vars ${SECRET} and rather than using env variables stores inside .octavia we will use secret manager that is inside aribyte? Thanks a lot for your response!!

Niclas Grahm

08/17/2022, 1:30 PM

Hi friends! When running octavia import all
, sources, destinations and connections are successfully generated as yaml files in my octavia project folder. Along which the configuration.yaml, there's also a state_***.yaml file generated which has the following info:

Copy code

configuration_hash: 96474a5f69b4844ba0f7dcabac12509ac3740b5dd4396cc0564d98e73d03ad6d
configuration_path: connections/local_postgres_local_json/configuration.yaml
generation_timestamp: 1660736874
resource_id: 67fed803-dd83-482f-aec7-745b517037dc
workspace_id: b72378fc-7a0e-4f48-9a4f-e0ba6dbd6cd7

I can't find anything in the docs about this file, what is it used for? Am I supposed to version control it? Thanks! 🙂

Piotr Sierkin

08/19/2022, 8:39 AM

Hello recently I was thinking what would be a nice addon to octavia cli Storing state on remote something like in terraform, I think it would be neat to have the possibility to store state on bucket rather than with vc

jan

09/07/2022, 9:42 AM

Hi there, any plans to include operations like triggering syncs in Octavia CLI as well? Seeing it autogenerates a python client but only partially covers all available API endpoints, I’m wondering if this is still on the roadmap or if we would need to kind of try to re-engineer Octavia CLI to allow for all available endpoints - which should already be covered in the autogeneration step.

Eka Mahadi

09/12/2022, 4:03 AM

Hello team, I noticed in some octavia commands, like

octavia apply

when updating a resource it will prompts user for confirmation before proceeding (

[y/N]

) I'm wondering whether octavia has an argument to

auto-yes / auto-confirm

, since I plan to use it in the ci/cd script

Krisjan Oldekamp

09/26/2022, 8:25 AM

Hi all! I was wondering if anyone already set up a workflow to automate version control of Airbyte setups? I'm managing multiple Airbyte instances / setups and I would like to automatically back up and version control changes that are made into the Airbyte interfaces. For example, fetch the different Airbyte configurations (using the Octavia CLI) once a day and check that into a Git repository, so we have version control and could use that to (manually) place back a configuration if necessary. Wondering if anyone already set up such a flow and how you've automated it? Thanks!

Dominik Mall

09/26/2022, 1:44 PM

Octavia CLI secrets Hi there, I’m currently struggling trying to properly pass secrets with Octavia. I have a self-hosted airbyte instance deployed on GKE via Helm. Using a BigQuery destination, if I apply the config with

hmac_key_secret: ${HMAC_SECRET}

and

hmac_key_access_id: ${HMAC_ID}

, and have both of those variables set in my terminal environment, the connection test fails - replacing the values in the UI with the actual ID/Secret causes it to pass. I also tried setting those variables in

~/.octavia

, but that didn’t work either. 1. Am I doing something obviously wrong here? 2. Is there a way to check what the actual value of the secrets are? (redacted configuration.yaml in thread)

zafar mahmood

09/29/2022, 8:52 AM

Hi , can anyone give some hints how exactly the data-type binding works between different connectors ? for example the source stream i use is Oracle by default some columns have datatypes of VARCHAR and INTEGER. And The destination is Postgres by default the datatype for majority of columns by AirByte are TEXT and FLOAT. Can we actually change this to VARCHAR (255) and Int8 etc ??. I tried changing it via Octavia-CLI in the stream information, (TEXT -> VARCHAR) but it overwrites to jsonb object. (edited)

Oliver Meyer

09/30/2022, 12:51 PM

Hi 🙂 I'm trying to understand if/how we can use octavia in a CI/CD pipeline and I'm confused about secrets management. Reading the conversations above I understand that secrets should be stored in

~/.octavia

if running octavia locally. Are we supposed to write secrets to the same file in whatever container will run the octavia commands? The readme mentions that env vars in configuration files are expanded, but that doesn't seem to be the case. Is the doc up to date?

Henri Blancke

10/13/2022, 1:56 PM

👋 I am trying out

octavia-cli

and noticed that when applying an update the all/affected streams will be reset. In the UI you have the ability to bypass that, is there a way to instruct octavia not to automatically reset all or affected streams?

Henri Blancke

10/18/2022, 7:44 PM

👋 as of Airbyte

v0.40.10

I'm getting the following error when trying to import any connection using

octavia-cli

. Anything I should change for octavia to support +v0.40.9? Everything seems to work in

v0.40.9

. Thank you!

Copy code

Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/check_context.py", line 91, in wrapper
    f(ctx, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 156, in connection
    click.echo(import_connection(ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], resource))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 121, in import_connection
    managed_resource, state = resources.factory(api_client, workspace_id, new_configuration_path).manage(connection_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 829, in factory
    return Connection(api_client, workspace_id, raw_configuration, configuration_path)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 273, in __init__
    self.configuration = self._deserialize_raw_configuration()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 601, in _deserialize_raw_configuration
    self._check_for_legacy_connection_configuration_keys(configuration)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 760, in _check_for_legacy_connection_configuration_keys
    self._check_for_wrong_casing_in_connection_configurations_keys(configuration_to_check)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 779, in _check_for_wrong_casing_in_connection_configurations_keys
    for stream in configuration_to_check["sync_catalog"]["streams"]:
KeyError: 'sync_catalog'

Dusty Shapiro

10/25/2022, 2:56 PM

I am curious; Is there value in using Octavia CLI to export workspace JSONS to check into version control if we’re deploying Airbyte with an external app DB?

Julien F

10/31/2022, 8:55 AM

Is there any way to reset a connection from the command line? I want to create a script that resets my staging environment, and I would like to do the same thing as the “reset your data” button?

lucien

11/04/2022, 10:10 AM

Hello Airbyte team A question regarding octavia, is it possible to disable the automatic data reset ? I would imagine something like

octavia apply --no-reset

It’s very important for us to be sure that our stream is not reset because we start to accumulate an history of all our updates in our source database and we really don’t want to lose it

Volodymyr Rudyi

11/15/2022, 10:42 AM

Hi! I have a couple of custom sources and added them through the Airbyte UI

settings >> +New connector

Can I generate them via Octavia CLI?

lucien

11/17/2022, 10:09 AM

Hi we really need this feature. Is it possible to merge this one ? https://github.com/airbytehq/airbyte/pull/18337

Matt Palmer

11/17/2022, 1:31 PM

Is there any documentation on how to implement Octavia for k8s deployments of Airbyte?

Dheeraj Pranav

11/23/2022, 7:22 AM

is there any issue while uploading data back to airbyte using

octavia apply

for creating connections on airbyte-UI. due to geography paramerter in configuration.yaml it's raising.

Igor

11/28/2022, 1:06 PM

Guys what is the best approach to deploy updated configurations from local machine to production?

Mateusz Kijewski

12/12/2022, 6:18 PM

Hello Everyone, I’m evaluating usage of Airbyte for our use-case. Is it possible to set up connection with custom source using Octavia CLI?

Tobias Löfgren

12/13/2022, 8:39 AM

Hi everyone! I am currently working with Octavia-CLI to redeploy our airbyte instance. I am following the tutorial how to set it up and creating sources and destinations is no issues at all. However, my stored passwords in my local ~/.octavia file are not exported into my airbyte instance. Does anyone know what might be failing? Airbyte instance: Version 0.40.23, Running with docker on a GCP Virtual Machine Octavia Cli: Installed with docker run, according to this My ~/.octavia file includes:

AIRTABLE_API_KEY="password"

The configuration file for airtable inclues:

tables: [Reported Time] # REQUIRED

api_key: ${AIRTABLE_API_KEY} # SECRET

base_id: appWvfaXxrKlmO7aP # REQUIRED

When I run octavia apply, it recognises that I have updated my secret password and updates the configuration. However, when I go into the airbyte UI and look at airtable it says ‘Invalid API key’ meaning the environment variable haven’t been applied

Simon Späti

12/19/2022, 2:03 PM

Cross-posting / Blog: An excellent addition to Octavia CLI is dagster if you want to set up your Airbyte instance with Python or need to act on external factors. Dagster makes it possible.

Hevandro Da Conceição Veiga

12/22/2022, 1:12 PM

Hi everyone! I'm new to airbyte and trying to use Octavia-CLI to automate some tasks. I got this error when trying to get connections. Anyone can help? Airbyte version: 0.40.17 Octavia-cli: Newest (0.40.26? How can I see octavia-cli version? )

Copy code

$ octavia --airbyte-url <http://localhost:8001> get connection 11cc95aa-b536-47b8-bffe-608e924d7a58
🐙 - Octavia is targetting your Airbyte instance running at <http://localhost:8001> on workspace 49658b56-4097-4ce5-bf55-a036722e8027.
Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/get/commands.py", line 97, in connection
    click.echo(get_json_representation(ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], Connection, resource))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/get/commands.py", line 67, in get_json_representation
    return resource.to_json()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/get/resources.py", line 148, in to_json
    return json.dumps(self.get_remote_resource().to_dict())
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/get/resources.py", line 138, in get_remote_resource
    return self._find_by_resource_id()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/get/resources.py", line 129, in _find_by_resource_id
    return self._get_fn(self.api_instance, self.get_payload)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api/web_backend_api.py", line 708, in web_backend_get_connection
    return self.web_backend_get_connection_endpoint.call_with_http_info(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 849, in call_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 410, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 225, in __call_api
    return_data = self.deserialize(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 326, in deserialize
    deserialized_data = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
TypeError: _from_openapi_data() missing 3 required positional arguments: 'schema_change', 'notify_schema_changes', and 'non_breaking_changes_preference'

Hrvoje Piasevoli

12/23/2022, 3:47 PM

Hi, i have upgraded airbyte yesterday to 0.40.25 using helm charts, deployment on GKE, external cloud SQL postgres. Octavia cli seems to be incompatible with the recent changes regarding field selection, import connection is broken, apply connection also... Any reported known issues regarding this?

Dimitri Grisard

12/30/2022, 11:18 AM

Hello 👋 I wanted to create a github action that deploys my sources, destinations & connections but with the local state system it's impossible? Or did I miss something?

Benoit Hugonnard

01/09/2023, 10:14 AM

Hello 👋 I have spent quite some time using Octavia-Cli and while it’s very helpful, I’m experiencing some issues : • The first one is already documented here : issues with the 4 connection fields. When I put them in configuration, I got TypeError and when I don’t put them I have to accept “diff” for every connections on apply. ◦ schema_change ◦ notify_schema_changes ◦ non_breaking_changes_preference ◦ geography • The second one is when I try to block the source/destination versions on the file, it will still try to use the latest available of the docker. ◦ For Snowflake Destination for example, even if I override the definition_version of the config file with

0.4.38

it will still run with

0.4.41

, is there a way to configure the version we want for each source/destination ? Do you have any idea ?

Claudio Cavallo

01/10/2023, 7:33 PM

Hello guys, when I get a connection I receive some HTML code

Claudio Cavallo

01/10/2023, 7:33 PM

airbyte/octavia-cli:0.40.27