Title
#advice-data-transformation
s

Slackbot

06/20/2022, 10:01 AM
This message was deleted.
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 12:23 PM
Hi Eugene, thank you for sharing this! Which version of Airbyte are you using? Could you also share the version of the source-postgres connector and destination-bigquery connector? Do you mind opening a topic on our forum sharing this problem? It would help centralizing the discussion with our engineering team.
12:23 PM
Feel free to share the reply from Google support too 🙏🏻
Eugene Krall

Eugene Krall

06/20/2022, 12:31 PM
• Airbyte version is 0.33.11-alpha Can't find any information on connector versions from the interface. It also looks like Airbyte created all those VM instances. Is this technically possible? And if so, what's the reason behind it?
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 12:32 PM
What is your deployment method? Docker or kubernetes?
Eugene Krall

Eugene Krall

06/20/2022, 12:33 PM
docker
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 12:34 PM
Airbyte is not triggering creation of VM on your cloud account. I'm not sure this is an Airbyte-related problem. I'd suggest you deep dive a bit in GCP audit logs to understand who created these instances.
Eugene Krall

Eugene Krall

06/20/2022, 12:37 PM
Got it. I will. I just need to know it's impossible for airbyte to created new VM instances so I can get it off the equation. But all those instances were created exactly the day I launched a bunch of syncs through Airbyte so it was an obvious place to look at
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 12:39 PM
it's impossible for airbyte to created new VM instances so I can get it off the equation
Airbyte itself does not provision any infrastructure. This why I asked for your deployment method, a badly configured K8S cluster could lead to unlimited node creation, but again this is not something managed by Airbyte.
12:45 PM
Is the service account associated with your Airbyte VM permitted to create GCP instances.
Eugene Krall

Eugene Krall

06/20/2022, 12:46 PM
Yes. I guess it is
12:47 PM
but can't be 100% sure. I deleted the service account cause we were getting charged even though all syncs were on pause since Friday
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 12:50 PM
Did you had the opportunity to connect to these instance and get a glimpse of what they were running?
Eugene Krall

Eugene Krall

06/20/2022, 12:53 PM
judging by the activity name it's somehow connected with data insertion
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 12:57 PM
Are you sure that your are not running on Kubernetes? I see
pod
occurrences in your screenshot
Eugene Krall

Eugene Krall

06/20/2022, 1:09 PM
We are running Airbyte version on our own servers and we deployed it using Docker. We synchronise tables in our MongoDb and MySQL databases with BigQuery using Bigquery connectors. We don't use Kubernetes as far as I know
1:11 PM
We followed these instructions when deploying it, didn't change anything apart from what is required in the instructions: https://docs.airbyte.com/deploying-airbyte/local-deployment
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 1:35 PM
For the costs you mentionned, are they BigQuery costs or compute instance costs?
1:36 PM
We are running Airbyte version on our own servers and we deployed it using Docker.
Yes sorry the pod probably refer to internal Google namespace.
1:42 PM
Ok, you need to find which service account created the VM then.
Eugene Krall

Eugene Krall

06/20/2022, 1:47 PM
Our own syncs doesn't utilize this from what I can see - we simply upload CSV files.
1:51 PM
that's the only operation I see being performed on all those instances
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 1:56 PM
I'll triple check with our technical team but I can't find explicit reference to this kind of operation in our repo. It would help if you could get the IP address or any more details about these operations.
Eugene Krall

Eugene Krall

06/20/2022, 2:00 PM
and looks like we don't get any activities of this type since the time we shot down Airbyte. But we still got charged, I assume maybe it was because the instances had been created and hadn't been shot down (my lame explanation)
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 2:10 PM
judging by the activity name it's somehow connected with data insertion
The activity you see in the screenshot is not data insertion but bulk creation of GCP instances. As far as I know it's not something required in Airbyte realm, neither by Airbyte platform itself or the source and destination connector you are using. I wrote to our technical team to make sure that my assertions are correct. From my standpoint I'm under the impression that the GCP instances on which you run Airbyte got compromised and someone was able to run this bulk creation of instances for malicious activity. I can't be 💯 sure of this if you don't get more details about what is (or was) running on the new GCP instances.
Eugene Krall

Eugene Krall

06/20/2022, 2:12 PM
Thanks. I'll try to get info from Google Cloud team. I'm not able to dig into it deeper since after disabling the billing I'm no longer allowed in compute engine dashboards.
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 2:14 PM
I don't have any context about your organization IAM management, but I'm under the impression the
sendpulse-backend
service account is not a default service account that GCP provisions for a new VM. Did you assign this existing
sendpulse-backend
service account to your Airbyte VM ?
Eugene Krall

Eugene Krall

06/20/2022, 2:19 PM
yes. I've created a JSON key for this account and specified it in Airbyte.
2:19 PM
This one was created manually
Marcos Marx (Airbyte)

Marcos Marx (Airbyte)

06/20/2022, 2:25 PM
If you open the activity are you able to see the IP generated request?
Eugene Krall

Eugene Krall

06/20/2022, 3:28 PM
I can only see IPs of VM instances
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 3:30 PM
You need to consume Audit logs from GCP to get more details about what happenned on your account.
4:15 PM
👏🏻 For figuring this out and I'm sorry for you being a target. The only certitude you have is that your service account is probably compromised and you should renew it asap.
Eugene Krall

Eugene Krall

06/20/2022, 4:17 PM
I've deleted it earlier today. Should be more careful about this in the future. Hopefully I will be able to arrange a refund with Google Cloud
Augustin Lafanechere (Airbyte)

Augustin Lafanechere (Airbyte)

06/20/2022, 4:18 PM
I would suggest you reconnect to your original airbyte instance VM and try to check what command were run and try to find authentication log to check if someone from the outside of the organization got a console access to this VM.
4:25 PM
Even though it's not accessible to the outside world and runs locally on our servers with access through a private VPN
If you are confident about your networking set up you should double check the service account file did not leak. You might have committed it on a public repo?
4:32 PM
I'd suggest to remove all your organization related information from this thread too (all your screenshots