https://linen.dev logo
Join Slack
Powered by
I’m curious if anyone else who deploys Airbyte via...
# replication-troubleshooting
d
I’m curious if anyone else who deploys Airbyte via Helm chart to K8s is able to use the Octavia CLI alongside it? Edit: Perhaps by adding it to the 
extraContainers
array in the chart values?
✍️ 1
u
@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2840 to ensure timely resolution!
m
Just for me to understand, you can't access your deployment with octavia cli? Octavia uses the Airbyte APIs to make any changes, so if Airbyte server is accessible it will be able to do changes, you may need to pass the argument 
--airbyte-url
so that the URL for the server is known to it.
Octavia-cli is not really "deployed", it is an application that runs on docker, and shoud be used on the developer machine or DevOps pipeline
d
Airbyte (for us) is deployed via Helm to a K8s cluster. I would like to use the CLI to extract configurations to check into version control. I would need the ability to use the CLI in the deployed environment to do so.
m
If you can access Airbyte UI from your machine, Octavia-cli will work on your machine, it does not need to be deployed on the environment, it is not part of Airbyte, it is it's own thing. Sorry if I am not understanding the issue
d
We can access the UI, but Airbyte is deployed to a dev/prod environment and accessed through the webapp ingress.
Nothing is running locally
Which is why i need a container with Octavia deployed to each environment to interact with the container in it’s respective environment. Maybe I’m confused as well 🤷
m
Airbyte uses Airbyte APIs to do it's thing, for example: 
octavia-cli --airbyte-url=<http://airbyte.mycompany.com> import source <UUID>
will import the source configuration to your machine, so you can commit it on git for example. You use this command on your machine, it doesn't need to be deployed, octavia is basically a bash command that you run on your machine
the URL that you use is the same to access Airbyte UI
d
I get that, but none of the containers running Airbyte exist locally for me. I guess I need to just deploy an empty container to pull the Octavia docker image into and run it in parallel to my deployed containers.
m
You don't need to do that, octavia doesn't need to Airbyte to exist locally
d
Ok, will give it a second shot. Thanks!
Copy code
Error: Could not reach your Airbyte instance, make sure the instance is up and running and network reachable: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Sat, 15 Oct 2022 01:30:38 GMT', 'Content-Type': 'application/json', 'Content-Length': '2', 'Connection': 'keep-alive'})
HTTP response body: {}
My guess this is because of the Google Proxy we use in the web ingress
m
Yes! But there is a file that you can configure for the authentication, I forgot the name right now...
d
Appreciate the info, but with how we use Google OAuth, not sure headers will work. I’ll check with my infra team, but fairly certain I’ll need to deploy a container to K8s with octavia image built in it so as to not have to authorize into the server container
h
Hey since it is dealing with the API if you can run octavia cli inside a container where the Security layer is not there you should be able to run octavia. But you don't need to run a seperate container for that unless that is a needy requirement
d
Thanks @Harshith (Airbyte), I’ll give that a shot. If not a separate container, would that mean running it in the webserver pod, for example?
s
Hey @Dusty Shapiro, sorry for the delay in response. Were you able to get this working? Feel free to post more questions in the channel 🙂
d
Technically, yes. I deployed a K8s job to the same cluster and am able to communicate with the Airbyte pod. The only downside is the Job isn’t persistent and gets wiped on deploys. That sounds like a personal infra problem, though, it’s just not a priority to update at the moment.
c
Hi @Dustin Shapiro, did you manage to solve this?
5 Views
Open in Slack
PreviousNext
Powered by