*Is this your first time deploying Airbyte*: Yes *...
# troubleshooting
r
Is this your first time deploying Airbyte: Yes OS Version / Instance: Mac OS 11.6.4 Memory / Disk: 32Gb / 500Gb Flash Storage Deployment: Docker Airbyte Version: 0.35.53-alpha Source name/version: Postgres 12.8 on GCP Cloud SQL Destination name/version: BigQuery Step: Setting new connection, source / On sync Description: I am evaluating Airbyte. I have setup a new connection successfully but it does not retrieve any streams (~tables) for me to select from the source. It does not give any error while setting up the connection. Running a sync does not sync any tables. I am trying to connect to Postgres source running on GCP Cloud SQL via a cloud proxy. If I connect to Postgres source running locally on my Mac it retrieves the schema (~tables) from the source fine for me to select via UI. Attached are logs and screenshot. Please help !!
r
I'm also keeping an eye out to a solution for this!
s
Hey there -- I've actually done this exact setup before in a prod env and have it running for multi TB syncs so might be able to help you. Have you tried running this against a GCE vm instead of on Mac?
r
Hi @Saif Abid To answer your questions: • Have you tried connecting without cloud sql proxy and had a successful sync? Mac -> PG CloudSQL direct Yes. This works. Running direct with Postgres on my local w/o proxy works • Have you tried running this against a GCE vm instead of on Mac? GCE -> Cloud SQLProxy -> PG CloudSQL
c
One other than that is different about @Rajnish Malik’s setup is that locally we have a privileged user, but on CloudSql we have a provisioned user that's been setup with read-only permissions. When connecting with those credentials via dbeaver and other IDEs, we do see the full list of tables, but I wasn't sure if Airbyte could be trying to access the schema info in a different way that the credentials might not have access to. Is there anything that describe the exact SQL or PG commands that Airbyte is using to access the schema?
I think I found the problem, and I can submit an issue in git. I think the query in the postgres source connector is not checking for nested privileges. https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/s[…]ava/io/airbyte/integrations/source/postgres/PostgresSource.java for example, if I have a username called foo, and he has been granted the foo_role , the query in the above code is going to fail, as it's grantee in the table_privileges table is only going to list foo_role, when the query is filtering on grantee = <username> does this make sense?