Octavia CLI secrets Hi there I m currently struggling tryin Airbyte #help-api-cli-orchestration

Octavia CLI secrets Hi there, I’m currently strugg...

Dominik Mall

09/26/2022, 1:44 PM

Octavia CLI secrets Hi there, I’m currently struggling trying to properly pass secrets with Octavia. I have a self-hosted airbyte instance deployed on GKE via Helm. Using a BigQuery destination, if I apply the config with

hmac_key_secret: ${HMAC_SECRET}

and

hmac_key_access_id: ${HMAC_ID}

, and have both of those variables set in my terminal environment, the connection test fails - replacing the values in the UI with the actual ID/Secret causes it to pass. I also tried setting those variables in

~/.octavia

, but that didn’t work either. 1. Am I doing something obviously wrong here? 2. Is there a way to check what the actual value of the secrets are? (redacted configuration.yaml in thread)

Dominik Mall

09/26/2022, 1:45 PM

Copy code

resource_name: BigQuery p-foundation me
definition_type: destination
definition_id: 22f6c74f-5699-40ff-833c-4a879ea40133
definition_image: airbyte/destination-bigquery
definition_version: 1.2.3
configuration:
  dataset_id: <redacted>
  project_id: <redacted>
  loading_method:
    method: GCS Staging
    credential:
      credential_type: HMAC_KEY
      hmac_key_secret: ${HMAC_SECRET}
      hmac_key_access_id: ${HMAC_ID}
    gcs_bucket_name: <redacted>
    gcs_bucket_path: data_sync/me_prod
    keep_files_in_gcs-bucket: Delete all tmp files from GCS
  dataset_location: EU
  transformation_priority: interactive
  big_query_client_buffer_size_mb: 15

Matt Webster

09/27/2022, 12:38 AM

Hi Dominik, for what it’s worth secret interpolation is not working for me either. I haven’t looked at GitHub as to whether anything has been reported but it’s possible there’s just a bug here

Matt Webster

09/29/2022, 1:59 AM

Hi @Augustin Lafanechere (Airbyte), sorry to intrude but at least 2 of us are having trouble getting secret interpolation to work. I’m setting variables in zsh (confirm with printenv) then running octavia apply in the same session. But the content seems to be sent as a string. At least it doesn’t work after the apply but if I put the value in manually in the Airbyte Web UI it works fine. Any ideas?

Copy code

resource_name: Data Warehouse
definition_type: destination
definition_id: 25c5221d-dce2-4163-ade9-739ef790f503
definition_image: airbyte/destination-postgres
definition_version: 0.3.24
configuration:
  ssl: false
  host: localhost
  port: 5432
  schema: public
  database: postgres
  username: postgres
  password: ${DB_PASSWORD}
  tunnel_method:
    tunnel_method: NO_TUNNEL

Augustin Lafanechere (Airbyte)

09/29/2022, 8:53 AM

hey @Matt Webster the secrets should be set in the

~/.octavia

file if you are running

octavia

commands locally

Augustin Lafanechere (Airbyte)

09/29/2022, 8:55 AM

hey @Dominik Mall just to confirm, you are running

octavia

locally but you are targetting an Airbyte instance on GKE right?

Dominik Mall

09/29/2022, 8:55 AM

Hi @Augustin Lafanechere (Airbyte), yes that is correct!

Matt Webster

09/29/2022, 5:46 PM

Thanks for the quick response. It’s still not working for me. My ~/.octavia file is now…

Copy code

OCTAVIA_ENABLE_TELEMETRY=True
DB_PASSWORD=[REDACTED]

…but it still doesn’t work when I apply. When I fix in the UI it is fine.

Matt Webster

09/29/2022, 5:47 PM

Any ideas how to troubleshoot? I looked at the code and couldn’t figure it out. It’s not obvious where the .octavia file gets loaded.

Matt Webster

09/29/2022, 7:09 PM

OK. I’ve got it working now! Unfortunately, I’ve been resetting configs and containers so much I’m fuzzy on what I actually did to make it work. But environment variables are in ~/.octavia now and they work in the YAML. It’s this line in the docs that I overlooked: “…whose values are mapped to the octavia container’s environment variables.” Thanks for the help.

Matt Webster

09/29/2022, 7:11 PM

As an enhancement, I’d love to be able to override .octavia settings with environment variables. If I get a break maybe I’ll do a PR. But either way I’m unblocked!

Matt Webster

09/29/2022, 7:45 PM

To elaborate a bit more…I’m starting my local source and target DB containers using a .env file and Docker Compose. Would love to point octavia to the same file (or set via VS Code launch) so I don’t need to manage these in separate places.

Augustin Lafanechere (Airbyte)

09/30/2022, 1:42 PM

@Matt Webster the

octavia

command is indeed an alias to a more complex

docker run

command i . The

.octavia

file is used as the

env-file

to set the env vars of the

octavia

container:

docker run -i --rm -v $(pwd):/home/octavia-project --network host --env-file ${OCTAVIA_ENV_FILE} --user $(id -u):$(id -g) airbyte/octavia-cli:0.40.6

Augustin Lafanechere (Airbyte)

09/30/2022, 1:46 PM

You could add

-e

options like

-e MY_SCRET_VAR=${MY_SECRET_VAR_SET_IN_HOST_ENV}

to the alias

Matt Webster

10/04/2022, 10:09 PM

Haven’t tried these yet, Augustin, but wanted to say it’s on my list! And the important thing is that we’re unblocked and don’t need to store secrets in source control.

73 Views

Open in Slack

Previous Next