Title
#shameless-plugs
koji matsumoto

koji matsumoto

11/16/2021, 1:19 PM
Hi everyone 👋 I wrote a blog about the production operation of Airbyte on GCP. It's a content that implemented a monitoring and config backup using GCP resources.https://dev.to/kojim/starting-small-airbyte-on-gcp-1j19
Michel

Michel

11/16/2021, 10:37 PM
Sooooo cool!!!!
10:37 PM
thank you for sharing
Jonas Bolin

Jonas Bolin

11/17/2021, 1:25 PM
@koji matsumoto would you share your cloud function for backing up the config yaml?
1:25 PM
Great article btw!
koji matsumoto

koji matsumoto

11/17/2021, 2:10 PM
I'll share only the excerpt of the main process. It's written by python as follows:
from google.cloud import storage

# export Airbyte config
req = urllib.request.Request(AIRBYTE_CONFIG_ENDPOINT, method='POST')
with urllib.request.urlopen(req) as data:
    conf_data = data.read()

# upload Airbyte config to GCS
client = storage.Client()
bucket = client.get_bucket(BUCKET)
blob = bucket.blob(GCS_PATH)
blob.upload_from_string(conf_data, content_type='application/x-gzip')
Jonas Bolin

Jonas Bolin

11/17/2021, 2:22 PM
Cheers!@koji matsumoto If you don't mind another question from a GCP beginner: when calling the Airbyte endpoint from a Cloud Function within the same GCP project. Do you use the IP of the Compute instance, or how you you access the API?
Max Krog

Max Krog

11/17/2021, 8:15 PM
@Jonas Bolin without having the full context (planning on reding the post tomorrow): Yes that’s basically it. To make it secure you can put the CF and VM on the same VPC.
koji matsumoto

koji matsumoto

11/18/2021, 12:16 AM
@Jonas Bolin Yes exactly, I made Cloud Functions access to VPC internal IP address attached to compute instance. For that I'm using Serverless VPC access for connecting Cloud Functions and VPC.
12:19 AM
Thank you for your question, I forgot to describe about that point in the article. I've updated the post!