https://linen.dev logo
Join Slack
Powered by
If i run airbyte on EC2 And configure a remote dat...
# replication-troubleshooting
r
If i run airbyte on EC2 And configure a remote database 1. Can i treat the instance as ephemeral or is there data that needs to be stored there? Its only for remote sources/connections. 2. If i deleted it and re-created a few days later would it start from where it last knew it was and continue without any issues once the next sync was ran for a connector? Mainly thinking of incremental sync's
✍️ 1
Also when trying to run an external db on rds temporal seems to fail to connect to rds:
Copy code
+ temporal-sql-tool --plugin postgres --ep <http://airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com|airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com> -u superuser -p 5432 create --db temporal
2022-10-12T22:53:36.777Z        ERROR   Unable to create SQL database.  {"error": "unable to connect to DB, tried default DB names: postgres,defaultdb, errors: [pq: no pg_hba.conf entry for host \"10.1.102.94\", user \"superuser\", database \"postgres\", SSL off pq: no pg_hba.conf entry for host \"10.1.102.94\", user \"superuser\", database \"defaultdb\", SSL off]", "logging-call-at": "handler.go:97"}
+ temporal-sql-tool --plugin postgres --ep <http://airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com|airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com> -u superuser -p 5432 --db temporal setup-schema -v 0.0
2022-10-12T22:53:36.797Z        ERROR   Unable to connect to SQL database.      {"error": "pq: no pg_hba.conf entry for host \"10.1.102.94\", user \"superuser\", database \"temporal\", SSL off", "logging-call-at": "handler.go:52"}
+ temporal-sql-tool --plugin postgres --ep <http://airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com|airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com> -u superuser -p 5432 create --db temporal_visibility
2022-10-12T22:53:36.821Z        ERROR   Unable to create SQL database.  {"error": "unable to connect to DB, tried default DB names: postgres,defaultdb, errors: [pq: no pg_hba.conf entry for host \"10.1.102.94\", user \"superuser\", database \"postgres\", SSL off pq: no pg_hba.conf entry for host \"10.1.102.94\", user \"superuser\", database \"defaultdb\", SSL off]", "logging-call-at": "handler.go:97"}
+ temporal-sql-tool --plugin postgres --ep <http://airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com|airbyte-staging-01.cv0s9my58zow.us-east-2.rds.amazonaws.com> -u superuser -p 5432 --db temporal_visibility setup-schema -v 0.0
2022-10-12T22:53:36.842Z        ERROR   Unable to connect to SQL database.      {"error": "pq: no pg_hba.conf entry for host \"10.1.102.94\", user \"superuser\", database \"temporal_visibility\", SSL off", "logging-call-at": "handler.go:52"}
I only allow SSL connections... is that the issue? If so how can i configure it to connect over ssl?
nvm needed more env vars:
Copy code
- SQL_TLS=true
      - SQL_TLS_DISABLE_HOST_VERIFICATION=true
      - SQL_TLS_ENABLED=true
is this supported or explained anywhere or should i make a pullr equest to add these options to the docker-compose deploy?
u
@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2791 to ensure timely resolution!
s
1. Logs are by default stored in the docker containers so you may lose that data, unless you have external logging set up 2. Incremental sync tracking is stored in the destination tables, so as long as those aren't disrupted Airbyte should be able to operate as expected We do not have those env vars documented as far as I know, please submit a Github issue for this! If you're willing to submit a PR that would be great as well!
Open in Slack
PreviousNext
Powered by