No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

hi. here's an example to use cloud storage <https://docs.ploomber.io/en/latest/api/_modules/clients/ploomber.clients.GCloudStorageClient.html#ploomber.clients.GCloudStorageClient>

for big query, are you looking to run a query and dump the results into a local file or do you want to write SQL scripts that create new tables?

we're still working on improving these examples so please share your feedback!

my use case - read a kafka topic, store output to GCS, read GCS and update bigquery table after flattening it, transformation using SQL and create a materialized view.

I saw the documentation. I was hoping you have a github repo of working example of using Bigquery and GCS.

ah. we don't have a full example. thanks for the suggestion! I think it makes sense to add one. Also for AWS

but feel free to ask any questions here as you make progress with the gcloud integration

<@U0164GZJ5EK> - here is the error I’m getting. ModuleNotFoundError: An error occured when trying to import dotted path ‘clients.gcs’: No module named ‘clients’

do you have a `clients.py` file in the same folder as your `pipeline.yaml` ?looks like it cannot find it

thx. now it is ModuleNotFoundError: No module named ‘google’

probably missing the g cloud storage package? <https://pypi.org/project/google-cloud-storage/>

thx got it working. is there a way to parameterize clients.py? hate to use path hardcoded in .py file.

image.png

in your case it'd be something like:
```File:
  dotted_path: clients.gcs
  name: value```

hi <@U0370FZQZ7E>! I was looking for ideas for our next blog post and I decided to write something on using ploomber with google cloud :slightly_smiling_face: since you're working on setting this up, I thought of sharing the <https://github.com/ploomber/projects/tree/master/templates/google-cloud|code example I'm working on> - right now, it connects to bigquery, but I'll add the cloud storage stuff too. still work in progress but feel free to share your feedback

Do you plan to use sqlalchemy client or DBAPIClient or native BigQuery Client?

DBAPIPClient is a wrapper for the native client. using the sqlalchemy client has a few extra features, but both work fine