I have a psuedo-random question. In general, how does Airbyte handle APIs with strict rate-limits? Is this something we would configure or is it deep in the connector somewhere?

not sure if somebody is already looking into it.

What I have faced right now: I started to get a lot of errors like next

airbyte-temporal    | {"level":"error","ts":"2021-03-24T16:09:19.365Z","msg":"Operation failed with internal error.","service":"matching","error":"UpdateTaskQueue failed. Failed to start transaction. Error: context deadline exceeded","metric-scope":38,"logging-call-at":"persistenceMetricClients.go:804","stacktrace":"<http://go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160|go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160>"}
airbyte-temporal    | {"level":"error","ts":"2021-03-24T16:09:19.365Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","store-operation":"update-task-queue","error":"UpdateTaskQueue failed. Failed to start transaction. Error: context deadline exceeded","logging-call-at":"taskReader.go:166","stacktrace":"<http://go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:166|go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:166>"}
airbyte-scheduler   | 2021-03-24 16:09:23 INFO i.a.s.JobRetrier(run):53 - {workspace_app_root=/tmp/workspace/scheduler/logs} - Running Job Retrier...

maybe thats because my Internet stopped working, but the main problem is next: CPU usage increased a lot ( and probably memory leak too ), so my pc almost got freezed. Is it possible to limit usage somehow? I suppose I am not the only one who can face such issue.

Even if I progress beyond that and try to add a connector internally, still no luck. Infinite wait. No matter the connector.

Hi Dev team, I have the following question: K8s deployment can only be done on the Main namespace, will you support multiple namespaces soon?

And, Everything has to be scheduled on the same node, will you support decoupling soon?

has anyone run into the case where a connector image is available locally, and airbyte cannot find it?

this is a random question: when someone with a fancy 16-inch macbook pro has time, can you run two build commands for me? 1.

CORE_ONLY=1 ./gradlew build

2.

./gradlew :airbyte-integrations:connectors:destination-redshift:test

I’m on a 2017 Macbook pro while I wait for the M1 Pro to come out. Wondering if upgrading to a better Mac for half a year or so is worth it

following up on how to support dates in type normalization: the format for dates in JSONSchema is fairly well defined • Defined in ISO8601 format • `"date-time"`: Date and time together, for example, 

2018-11-13T20:20:39+00:00

. • `"time"`: Time, for example, 

20:20:39+00:00

• `"date"`: Date, for example, 

2018-11-13

https://json-schema.org/understanding-json-schema/reference/string.html#dates-and-times

I have some questions around how stream names and schemas are handled in the Postgres source.

What’s the

JDBCDestination

class for? Nothing seems to use it

are the destination integration tests run as part of the general airbyte build?

On the demo I upgraded to the new version of the exchange rate api and I’m running into a weird catalog configuration error.

are people able to run

AbstractJdbcSourceStandardTest

in their intellij? works for me on gradle but not in intellij

can anyone think of an example of nested cursor fields? to power incremental sync, airbyte uses a field like

updated_at

, reading from the source records whose

updated_at

is greater than the largest

updated_at

from the previous sync. So far I have only encountered instances where these cursor fields are at the “top-level” i.e: not nested within the record. But I’m curious to understand how prevalent nested cursor fields are e.g:

date_info.updated_at

have we ever discussed moving off github actions onto a build provider that offers more CPU? my gut tells me part of the slowness on github action is the 2 CPU limit

hey we are trying to do csv (HTTPS) -> bigquery using

file

connection. There is a

number

field in the csv data and when we are pushing to

bigquery

this is taken as null. can someone help me with this ?

is anyone seeing slow image pull times when running the airbyte server on master? mine has gone from seconds -> minutes

For Google ads we are neglecting all parameters which have

fieldInclusions

this is not populating all important information. For reference link is the line where this is done. Any possible solutions for this ?

Any idea on when this will be merged? https://github.com/airbytehq/airbyte/pull/2757

anyone seen errors similar to this?

airbyte-scheduler   | 2021-04-12 12:21:46 ERROR i.t.i.w.Poller(lambda$new$0):70 - {} - Failure in thread Activity Poller taskQueue="SYNC", namespace="default": 1
airbyte-scheduler   | io.grpc.StatusRuntimeException: INTERNAL: Not enough hosts to serve the request
airbyte-scheduler   | 	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
airbyte-scheduler   | 	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
airbyte-scheduler   | 	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
airbyte-scheduler   | 	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollActivityTaskQueue(WorkflowServiceGrpc.java:2696) ~[temporal-serviceclient-1.0.4.jar:?]
airbyte-scheduler   | 	at io.temporal.internal.worker.ActivityPollTask.poll(ActivityPollTask.java:105) ~[temporal-sdk-1.0.4.jar:?]
airbyte-scheduler   | 	at io.temporal.internal.worker.ActivityPollTask.poll(ActivityPollTask.java:39) ~[temporal-sdk-1.0.4.jar:?]
airbyte-scheduler   | 	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:265) ~[temporal-sdk-1.0.4.jar:?]
airbyte-scheduler   | 	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:241) [temporal-sdk-1.0.4.jar:?]
airbyte-scheduler   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
airbyte-scheduler   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
airbyte-scheduler   | 	at java.lang.Thread.run(Thread.java:832) [?:?]

happens when I put load on the my system. e.g. run now I'm running Acceptance Tests as well as a full

./gradlew build

TIL about Jabba (https://github.com/shyiko/jabba) a version manager for Java. It provides a really easy way to manage java versions. Gonna run it for a few days and report back if there are any glitches. It might be an easy way for non-java engs to install Java if they ever need it when interacting with Airbyte

Anyone know how to fix this error when trying to build? A bit of searching suggests I have the wrong version of something somewhere. Using Java version 16, which appears to be the most recent version, and according to this, it's the version that corresponds to class version 60...

TIL: printing a struct that inherits an

AbstractQueue

(

struct.toString()

) without overriding

toString

actually empties the queue.. thanks @charles for the assist!

is there a reason an acceptance test case has different behaviour when run by itself vs when run as part of the test suite? seems like the various test cases, though ordered, do not rely on each other

Hi guys! Whether such a feature request as logging queries? I’ll try to explain with an example: Following onboarding steps I was creating a connection to postgres destination and caught an error message like this:

2021-04-14 20:39:11 INFO (/tmp/workspace/8ea53a05-4867-4639-9f70-744cf07739ea/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - org.postgresql.util.PSQLException: ERROR: permission denied for database test_db

but this message didn’t tell me anything about what specific query request was sent to the db when this error occurred. During debugged I saw that the problem is not in the wrong password or the connector is not working correctly, but in the following query:

CREATE SCHEMA IF NOT EXISTS public;

during this method call. My db user just doesn’t have permissions to create a schema. So, if user will able to see specific queries logs he will have possibility to catching troubles like this with easy way and it will more friendly user experience.

Hey we connected Google Ads to Bigquery and I can see that it's always pulling data from the start data rather than from where it left last time. Any idea on what to do ?

Does Airbyte have any plans to support webhooks, i.e. a callback that says

{"connection_id": 142, "status": "success"}

to some arbitrary web URL?

https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods (powered by https://gvisor.dev/) are related to the kube container isolation discussion we mentioned in the sync the other day

The master build is broken since I merged in a change with changes to the acceptance test that require us to release connectors supporting namespaces. I'm holding off connector releases to coincide with our general release next week. I figured there is minimal impact since it's the weekend. Apologies for the inconvenience. Please let me know if this blocks anyone