@kapa.ai how to connect to amazon dsp

When I create a new connection using a Salesforce source and a MySQL destination I get an error "An unknown error occurred. (HTTP 413)" when trying to save the new connection.

We are using Airbyte cloud to import

hubspot

contacts

to big query. We are using

incremenal| Append Dedup

First day we got almost all records. After that we are seeing missing records and duplicate records. Any one face this issue ?

This message was deleted.

@kapa.ai When a job is starting up a read or write pod, is the time it waits for the pod to start configurable?

This message was deleted.

@kapa.ai Why am I getting this error trying to connect a Postgres source?:

2024-09-04 23:44:03 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-04 23:44:03 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-04 23:44:03 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-04 23:44:03 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-04 23:44:03 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-04 23:44:03 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.b.s.SshTunnel$Companion(getInstance):423 Starting connection with method: NO_TUNNEL
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: false
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresSource(toSslJdbcParamInternal):923 VERIFY_CA toSslJdbcParam verify-ca
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-1 - Starting...
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-1 - Start completed.
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(close):349 HikariPool-1 - Shutdown initiated...
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(close):351 HikariPool-1 - Shutdown completed.
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.b.IntegrationRunner(runInternal):224 Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-09-04 23:44:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresSource(main):699 completed source: class io.airbyte.integrations.source.postgres.PostgresSource
2024-09-04 23:44:03 INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):231 - Checking for optional control message...
2024-09-04 23:44:03 INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):253 - Optional control message not found. Skipping...
2024-09-04 23:44:03 INFO i.a.c.ConnectorWatcher(run):134 - Writing output of decd338e-5647-4c0b-adf4-da0e75f5a750_60e4fa12-edf5-467a-8c17-af5b61a6aeee_0_check to the doc store
2024-09-04 23:44:04 INFO i.a.c.ConnectorWatcher(run):136 - Marking workload as successful
2024-09-04 23:44:04 INFO i.a.c.ConnectorWatcher(exitProperly):189 - Deliberately exiting process with code 0.
2024-09-04 23:44:04 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-09-04 23:44:04 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- END CHECK -----

I am getting below error when testing source from the UI

io.temporal.failure.ActivityFailure: Activity with activityType='RunWithWorkload' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=74fe0279-99f3-38c6-ba24-28ed677288fa, identity='1@airbyte-064-worker-5585fcbd5d-5js9l', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

@kapa.ai, what database inside postgres in airbyte should i backup?

@kapa.ai - how do I stop repeated webhook alerts that send every two minutes for the same job status?

what is the field "airbyte_generation_id" on raw table?

@kapa.ai, can I run 2 version of airbyte on my local under kind version of kubernetes?

@kapa.ai i follow the guidance of airbyte to create target destination in postgres, but got error State code: 42501; Message: ERROR: permission denied for schema airbyte_internal

@kapa.ai check_connection 504 on windows

my oss helm deployment on google cloud uses minio and am getting this error:

Cannot publish to S3: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. (Service: Amazon S3; Status Code: 507; Error Code: XMinioStorageFull

does this mean the minio image used has a free mini storage from s3 ?

@kapa.ai what is the use of %S in datetime format?

2024-09-05 071122 destination > Retrying langchain_community.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 16.0 seconds as it raised RateLimitError: Rate limit reached for text-embedding-ada-002 in organization org-hhuVUVP2FFjsS32XGgfT3lGn on requests per day (RPD): Limit 200, Used 200, Requested 1. Please try again in 7m12s. Visit https://platform.openai.com/account/rate-limits to learn more. You can increase your rate limit by adding a payment method to your account at https://platform.openai.com/account/billing.. 392 2024-09-05 071138 destination > Retrying langchain_community.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 20.0 seconds as it raised RateLimitError: Rate limit reached for text-embedding-ada-002 in organization org-hhuVUVP2FFjsS32XGgfT3lGn on requests per min (RPM): Limit 3, Used 3, Requested 1. Please try again in 20s. Visit https://platform.openai.com/account/rate-limits to learn more. You can increase your rate limit by adding a payment method to your account at https://platform.openai.com/account/billing.. Why I get Rate Limit Error? I have paid openai api key then why I get this error while syncing to pinecone from postgres

I am having problems testing a destination. The symptom is: An unknown error occurred. (HTTP 504) When I look at the ingress-nginx logs I can see the 504 error. It is matched by a 499 error in the airbyte-abctl-webapp logs:

kubectl logs -n airbyte-abctl deployment/airbyte-abctl-webapp | grep 499
10.244.0.18 - - [05/Sep/2024:07:22:05 +0000] "POST /api/v1/destinations/check_connection HTTP/1.1" 499 0 "<http://airbyte.lambert.upowr.internal:8000/workspaces/218abdf9-17ec-4c34-b96f-0b690f413601/destination/c9d91ec7-d42d-494e-b2b7-b7d1c1ff793f>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "10.11.13.203"
10.244.0.18 - - [05/Sep/2024:07:39:23 +0000] "POST /api/v1/destinations/check_connection HTTP/1.1" 499 0 "<http://airbyte.lambert.upowr.internal:8000/workspaces/218abdf9-17ec-4c34-b96f-0b690f413601/destination/c9d91ec7-d42d-494e-b2b7-b7d1c1ff793f>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "10.11.13.203"

If I look at the pods:

kubectl -n airbyte-abctl logs pod/ion-postgres-check-0fcc281e-462f-4f91-a676-5712a7d79eb3-0-tubbi

I see these error messages:

2024-09-05T07:38:30.492200279Z main INFO Loading mask data from '/seed/specs_secrets_mask.yaml
2024-09-05 07:38:32 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [k8s, cloud, worker-v2, control-plane, oss]
2024-09-05 07:38:33 INFO i.a.c.ApplicationKt(main):20 - Context started
2024-09-05 07:38:33 INFO i.a.c.ApplicationKt(main):21 - 2114198162.00 ns/exec (total: 2.11s, 1 executions)
2024-09-05 07:38:34 INFO i.a.c.ApplicationKt(main):28 - Sidecar created
2024-09-05 07:38:34 INFO i.a.c.ApplicationKt(main):29 - 1688729457.00 ns/exec (total: 3.38s, 2 executions)
2024-09-05 07:38:35 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-09-05T07:38:35.642324378Z pool-3-thread-1 ERROR Recursive call to appender SecretMaskRewrite
2024-09-05 07:38:35 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- START CHECK -----
2024-09-05 07:38:35 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-09-05 07:38:36 WARN c.a.l.CommonsLog(warn):113 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant.If you are using Java 9+, you will need to include javax.xml.bind:jaxb-api as a dependency.
2024-09-05 07:39:32 INFO i.a.c.ConnectorWatcher(run):87 - Connector exited, processing output
2024-09-05 07:39:32 INFO i.a.c.ConnectorWatcher(run):90 - Output file jobOutput.json found
2024-09-05 07:39:32 INFO i.a.c.ConnectorWatcher(run):96 - Connector exited with 0

Can you explain the root cause of this error please and what configuration change I might need to fix it? I have tried upgrading from a t3.large to t3.xlarge to double the system resources from 8GB to 16GB but this had precisely no effect on the symptoms reported. I have verified that the EC2 instance does have psql connectivity to the instance involve (although there are no reports in ANY of the logs about a failure to connect to postgres).

I am trying to make connection from MSSQL source to snowflake datawarehouse. While the source connection was success, I get the following error when sync was enabled and started. Using the same credentials, I can login and select the tables in CDC schema User SnowflakeCDC does not have access to CDC schema in the following databases: AMMT. This user can only be used in initial_only snapshot mode Any hints?

@kapa.ai when I try to set up Bing Ads source, this error is returned "WebFault("Server raised fault: 'Invalid client data. Check the SOAP fault details for more information. TrackingId: ec5dce6b-e3c3-4fe0-8284-b1add1656787.'")"

Hello, there was an error in my pyairbyte reading data about the Amazon sp api connector. My current airbyte connector version is 4.4.1. How should I solve this problem? The following is my code

Where could I get documentation of MSSQL connector version 0.36?

Hi, I have an API that accept a file as an input. Which solution is suitable for me to make a source for it in AIrbyte?

@kapa.ai I am adding a new parameter to an exisiting airbyte source. I have managed to get all the tests passing locally other than the Acceptance Test. is this one only expected to pass in Airbytes Github CI rather than locally?

@kapa.ai how to downgrade the helm version

@kapa.ai I have a cursor field in incremental sync which returns two differently formatted datetimes. So If i set any one as the cursor field format, the sync fails once it detects a record for the another format. How do I solve this in airbyte connector builder

@kapa.ai How can I specify image pull secret for temporal?

How to use pagination on connector builder using graphql request

Random “A stream status (null.<stream-name>) has been detected for a stream not present in the catalog” causes multiple retries and finally succeeds