Deploying on GCP - can this page be reviewed please? https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine/ I don't really know how to install Airbyte and I'm trying to install to learn more about it and .... • Step 3 there's no brew command needed? (because gcloud is already installed) • I gave up at Step 4 because I couldn't figure out how to make things work

Sure, I can help you review the steps to deploy Airbyte on a Google Cloud Platform (GCP) Compute Engine instance based on the information provided in the extracts. 1. Create a new GCP instance: You can follow the instructions provided by Google [here](https://cloud.google.com/compute/docs/instances/create-start-instance). 2. Set variables in your local terminal: You need to set the

PROJECT_ID

and

INSTANCE_NAME

variables. Replace

PROJECT_ID_WHERE_YOU_CREATED_YOUR_INSTANCE

with your actual project ID and

airbyte

with the name you've used for your instance.

bash PROJECT_ID=PROJECT_ID_WHERE_YOU_CREATED_YOUR_INSTANCE   INSTANCE_NAME=airbyte # or any other name that you've used

3. Install Google Cloud SDK and initialize the gcloud command-line tool: If you already have

gcloud

installed, you can skip the

brew install --cask google-cloud-sdk

command. You can initialize the

gcloud

command-line tool using

gcloud init

. 4. List all instances in your project: You can verify that your instance is created by listing all instances in your project.

bash gcloud --project $PROJECT_ID compute instances list

You should see the Airbyte instance you just created. 5. Connect to your instance: You can connect to your instance using the following command:

bash gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME

6. Install Docker on your VM instance: You can install Docker on your VM instance by running the following commands in your VM terminal:

bash sudo apt-get update   sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common   curl -fsSL <https://download.docker.com/linux/debian/gpg> | sudo apt-key add --   sudo add-apt-repository "deb [arch=amd64] <https://download.docker.com/linux/debian> buster stable"   sudo apt-get update   sudo apt-get install -y docker-ce docker-ce-cli <http://containerd.io|containerd.io>   sudo usermod -a -G docker $USER

7. Install

docker-compose

on your VM instance: You can install

docker-compose

on your VM instance by running the following commands in your VM terminal:

bash sudo apt-get -y install docker-compose-plugin   docker compose version

8. Close the SSH connection: You can close the SSH connection to ensure the group modification is taken into account by running

logout

. After these steps, you can proceed to install and launch Airbyte. If you encounter any issues, you can reach out to the Airbyte community on [Slack](https://slack.airbyte.com/). Please note that these instructions have been tested on a

Debian GNU/Linux 10

VM instance. For testing Airbyte, an

e2.medium

instance and at least 30GBs of disk per node are recommended. For deploying Airbyte in a production environment, a

n1-standard-2

instance is recommended. Source: [Deploy Airbyte on GCP](https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine)

I’m not too familiar with the environment that is exposed in the browser window but the guide is set up in such a way that it’s intended to be run from your local terminal, not the browser window. But yeah if you had installed gcloud in some other way you would just skip the brew step

Ok, I'll try that 😃

I would suggest doing things from your local terminal vs the browser window in general however, because that is the “supported” way

OK

no problem. It helps to understand how the gcloud interface works in general, the gcloud command line tool is a way for you to do gcloud related things from your local machine. You typically in sysadminy land don’t use these browser terminal windows for various reasons if you can avoid it, but they are there for you to have a quick and easy interface with the machine when you need to do something and aren’t hooked in via your local machine/gcloud

So if my laptop is Windows 11 then I just use "command" to get a "DOS" terminal and use that. That's if people actually know DOS these days 🤪

exactly. The typical workflow to do this is “install gcloud on my local machine” -> “authenticate to gcloud from my local machine” -> “gcloud commands reach out to gcloud and do things in gcloud, but it’s executed from my local machine” (or some ci). And in this specific case, what the commands are doing is “ask gcloud for the location of a remote ec2 instance” -> “use gcloud to ssh into that remote ec2 machine” -> “execute commands on that remote ec2 machine that install airbyte”

Awesome, thanks 😁

if you’re using windows, btw, you’ll probably want to use Windows Subsystem for Linux to get these commands working from cmd: https://learn.microsoft.com/en-us/windows/wsl/install Brew is a macos command (I believe it works on linux as well), but this guide is definitely framed in such a way that the commands are intended to be executed from a linux machine. So If you are using wsl it should just work

I can always switch to Mac as well

either way works, though we specifically do support windows users so the docs should probably be updated to use the windows equivalent as well, because by virtue of saying “use brew”, the docs assume a macos installation since you don’t typically use brew on linux machines or windows machines

Yeah, hence my post - I've never used Airbyte before, so I don't know how to install it. I'm trialling it to see if it solves a situation for my client. But if the installation process is too difficult to follow it makes installation hard

curious if the Marketplace would have solved your issue, e.g. a one-click airbyte installation. We debate internally how important it is to support one clicks, so we would be quite interested if you would have opted for that instead if it was well supported and available

I'm hoping that when it's installed and working that I have a browser interface that can be used to configure sources and destinations.

I’ll surface this feedback with the team (I am one of the engineers responsible for determining the support of one-click installations) 😄 To your original question, yes, once you spun it up with docker you would have a web interface. But then you have to handle networking to expose it externally.

I'm out of my comfort zone anyway, it's all good, just more stuff to learn 😉

if the purpose is just to set up something for demo purposes before sinking a bunch of time into spinning up OSS I would suggest using cloud if you can to trial it and then perhaps spinning up OSS if your client wants to proceed. Fundamentally the cloud product is just running a modified version of oss under the hood so the sync functionality is going to be roughly the same (minus that we support less connectors in cloud, but all the common ones are supported). I’m not sure what the requirements are though, (like does it have to be gcp, self hosted, etc) so it’s hard to give recommendations beyond that

Yeah, hence my other posts about understanding how credits are consumed

ah yeah, I cannot offer support there unfortunately, that is not my area of expertise 😄

At the moment the aim is about 1k records per day via an API. If that means we're using a fraction of a credit every day and a credit will last 1-2 months (or more) of loading then it's awesome

in the meantime, while your other pricing questions are answered, an easier way to play with airbyte without going straight to hosting it in the cloud (which can be a challenge, especially if you haven’t hosted things in the cloud before, and also airbyte is not a simple application either, there are a lot of moving gears in what it does), I would suggest following this guide and run some syncs on your local machine if that’s an option. Doing that takes the networking and remote instance stuff out of the equation https://docs.airbyte.com/quickstart/deploy-airbyte/ That one would have you using windows subsystem for linux (you are executing a .sh script)

OK, I'll try that as well

👍 that will probably be sufficient for you as long as you are running a somewhat decent machine. Airbyte isn’t suer beefy for resource requirements but it’s not lightweight either

The problem is - what's the best/cheapest way to get the API data into Snowflake for use with dbt and Tableau. Small client= low budget