https://linen.dev logo
#releases-and-early-demos
Title
# releases-and-early-demos
k

koji matsumoto

11/16/2021, 1:19 PM
Hi everyone 👋 I wrote a blog about the production operation of Airbyte on GCP. It's a content that implemented a monitoring and config backup using GCP resources. https://dev.to/kojim/starting-small-airbyte-on-gcp-1j19
👏 8
m

Michel

11/16/2021, 10:37 PM
Sooooo cool!!!!
thank you for sharing
j

Jonas Bolin

11/17/2021, 1:25 PM
@koji matsumoto would you share your cloud function for backing up the config yaml?
Great article btw!
k

koji matsumoto

11/17/2021, 2:10 PM
I'll share only the excerpt of the main process. It's written by python as follows:
Copy code
from google.cloud import storage

# export Airbyte config
req = urllib.request.Request(AIRBYTE_CONFIG_ENDPOINT, method='POST')
with urllib.request.urlopen(req) as data:
    conf_data = data.read()

# upload Airbyte config to GCS
client = storage.Client()
bucket = client.get_bucket(BUCKET)
blob = bucket.blob(GCS_PATH)
blob.upload_from_string(conf_data, content_type='application/x-gzip')
j

Jonas Bolin

11/17/2021, 2:22 PM
Cheers! @koji matsumoto If you don't mind another question from a GCP beginner: when calling the Airbyte endpoint from a Cloud Function within the same GCP project. Do you use the IP of the Compute instance, or how you you access the API?
m

Max Krog

11/17/2021, 8:15 PM
@Jonas Bolin without having the full context (planning on reding the post tomorrow): Yes that’s basically it. To make it secure you can put the CF and VM on the same VPC.
k

koji matsumoto

11/18/2021, 12:16 AM
@Jonas Bolin Yes exactly, I made Cloud Functions access to VPC internal IP address attached to compute instance. For that I'm using Serverless VPC access for connecting Cloud Functions and VPC.
Thank you for your question, I forgot to describe about that point in the article. I've updated the post!
2 Views