Hey team, while we all know that storage is cheap these days, there is no need to take a sledgehammer to crack a nut. We all know that (self hosted) Airbyte uses a lot of storage and I think we can agree that the recommendation of at least 30GBs of disk per node is the absolute lowest minimum. So my question. Did anyone ever look into how much disk space is needed on average per connector? I assume that that would be the easiest metric to determine how much storage is actually needed by your Airbyte instance. Or how do you approach this issue?

Thread sent successfully to Harvestr

Hey @Sebastian Brickel, we do not currently have benchmarking completed for Airbyte sources but it's a great idea to track this metric. We do currently have some open tickets around benchmarking, feel free to leave a thumbs up and comment: https://github.com/airbytehq/airbyte/issues/11719 https://github.com/airbytehq/airbyte/issues/17186 There are also tickets for specific connectors if you are interested in anything specific

Thank you very much @Sunny Hashmi (Airbyte)

Feedback was linked to Deployment UX

Linked feedback:

Hey team,

while we all know that storage is cheap these days, there is no need to take a sledgehammer to crack a nut.

We all know that (self hosted) Airbyte uses a lot of storage and I think we can agree that the recommendation of at least 30GBs of disk per node is the absolute lowest minimum.

So my question. Did anyone ever look into how much disk space is needed on average per connector? I assume that that would be the easiest metric to determine how much storage is actually needed by your Airbyte instance.

Or how do you approach this issue?