Hello I m new to the channel and hoping to get started with Airbyte #ask-community-for-troubleshooting

Hello, I'm new to the channel and hoping to get st...

Jeremy Owens

12/20/2021, 9:26 PM

Hello, I'm new to the channel and hoping to get started with working with Airbyte on k8s via Helm. Would anyone happen to have an example of a working set of

values

. I keep running into a problem where the server fails to start because it can't connect to the postgres installation.

✅ 1

[DEPRECATED] Marcos Marx

12/20/2021, 9:40 PM

@Jeremy Owens can you share the error logs you got?

Jeremy Owens

12/20/2021, 9:43 PM

Untitled.txt

Jeremy Owens

12/20/2021, 9:43 PM

I just used the stock values and the ci.sh file intially for this run, but I've tried numerous configurations

[DEPRECATED] Marcos Marx

12/20/2021, 9:46 PM

The worker is also crashing, can you get those logs too?

Jeremy Owens

12/20/2021, 9:47 PM

Untitled.txt

Jeremy Owens

12/20/2021, 9:49 PM

Service name looks right:

Copy code

~$ kubectl get svc -n airbyte
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
airbyte-minio                 ClusterIP   100.65.252.249   <none>        9000/TCP   14m
airbyte-postgresql            ClusterIP   100.68.98.44     <none>        5432/TCP   14m
airbyte-postgresql-headless   ClusterIP   None             <none>        5432/TCP   14m
airbyte-server                ClusterIP   100.64.242.139   <none>        8001/TCP   14m
airbyte-temporal              ClusterIP   100.71.246.213   <none>        7233/TCP   14m
airbyte-webapp                ClusterIP   100.69.119.137   <none>        80/TCP     14m
admin@ip-10-12-102-227:~$

Jeremy Owens

12/20/2021, 9:50 PM

Thanks for taking a look

Harshith (Airbyte)

12/21/2021, 10:31 AM

Hey @Jeremy Owens I can see that there is no airbyte-db pod created. Can you check why is that so ?

Jeremy Owens

12/21/2021, 1:27 PM

Is that not airbyte-postgresql ?

Jeremy Owens

12/21/2021, 1:46 PM

Making some progress. I inspected the temporal pod and saw this error:

Copy code

{"level":"error","ts":"2021-12-20T22:05:28.288Z","msg":"Operation failed with internal error.","service":"matching","error":"UpdateTaskQueue failed. Failed to start transaction. Error: pq: remaining connection slots are reserved for non-replication superuser connections","metric-scope":38,"logging-call-at":"persistenceMetricClients.go:804","stacktrace":"<http://go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160|go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160>"}
{"level":"error","ts":"2021-12-20T22:05:28.288Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/1","wf-task-queue-type":"Activity","store-operation":"update-task-queue","error":"UpdateTaskQueue failed. Failed to start transaction. Error: pq: remaining connection slots are reserved for non-replication superuser connections","logging-call-at":"taskReader.go:166","stacktrace":"<http://go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:166|go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:166>"}

Jeremy Owens

12/21/2021, 1:59 PM

I increased the maxConnections and got a bit further. This is what the server is doing now:

Copy code

2021-12-21 13:58:40 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...
2021-12-21 13:58:40 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$5):127 - Testing airbyte configs database connection...
2021-12-21 13:58:40 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):90 - Database available!
2021-12-21 13:58:40 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...
2021-12-21 13:58:40 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$5):127 - Testing airbyte configs database connection...
2021-12-21 13:58:40 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):90 - Database available!
2021-12-21 13:58:40 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...
2021-12-21 13:58:40 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$5):127 - Testing airbyte configs database connection...
2021-12-21 13:58:40 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):90 - Database available!
2021-12-21 13:58:40 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...
2021-12-21 13:58:40 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$5):127 - Testing airbyte configs database connection...
2021-12-21 13:58:40 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):90 - Database available!

Jeremy Owens

12/21/2021, 2:13 PM

In order to test the connection, I set up a ubuntu pod and installed psql

Copy code

root@ubuntu:/# psql -h airbyte-postgresql -U airbyte -d db-airbyte
Password for user airbyte: 
psql (12.9 (Ubuntu 12.9-0ubuntu0.20.04.1), server 11.14)
Type "help" for help.

db-airbyte=> \d+
Did not find any relations.
db-airbyte=> \d
Did not find any relations.
db-airbyte=>

Jeremy Owens

12/21/2021, 5:24 PM

Update: I resolved all of this by using

v0.33.12-alpha

instead of the existing master branch

6 Views

Open in Slack

Previous Next