good morning team Anyone managed to upload the airbyte with EKS - Fargate on Amazon can help me? I have volume related problem

Hello. Is there a way for connections between sources and sinks to be setup from config files as compared to strictly from the UI?

Hello, I am facing an issue with moving records from mongodb to elasticsearch. In the ES, the index are getting created, but the documents are not being inserted. It halts at

writing 1 records in bulk operation

Is there a way to rate limit Survey Monkey? Like if we have a 10k daily limit but only want airbyte to consume 8k.

Hello! I'm trying to use the 3PL Central source, but it isn't showing up as a connector. Do I need to add it as a source manually? https://docs.airbyte.com/integrations/sources/tplcentral

I am just now exploring Airbyte, and not sure which is the appropriate channel for this, and didn't find it in previous threads, so my apologies if I missed it. I have set up a few connections in AWS with S3 and RDS and I don't see any way to parameterize the source or destination to re-use the same connection but with different targets, i.e. different buckets or path in S3 or different table/views in RDS. I tried out the API and was able to clone and update new connections, but still had to hard code the path. I want to be able to set up, for example, an S3 destination for all my sources using the same bucket with the same creds, but a different folder path for each source, so say s3 bucket raw/postgres or raw/snowflake without having to create a new destination connection for each folder path. This would be a management nightmare if I have 100 connections all to the same bucket with just different paths. Am I missing something?

i have a problem : Caused by: io.airbyte.workers.WorkerException: Normalization Failed.

?

Hey all, I'm testing a simple flow from google sheets into MySQL....the connections work fine, but the execution keeps failing. Attached are the logs...any idea what is going on?

Hi Team- I am trying the google ads source connector with a custom query on CLICK_VIEW.I am getting the below error.I am not able to add a filter with segments.date as well.How to fix this error? google.ads.googleads.errors.GoogleAdsException: (<_InactiveRpcError of RPC that terminated with: status = StatusCode.INVALID_ARGUMENT details = "Request contains an invalid argument." debug_error_string = "{"created":"@1649897442.844915447","description":"Error received from peer ipv4172.217.24.42443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Request contains an invalid argument.","grpc_status":3}"

, <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INVALID_ARGUMENT details = "Request contains an invalid argument." debug_error_string = "{"created":"@1649897442.844915447","description":"Error received from peer ipv4172.217.24.42443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Request contains an invalid argument.","grpc_status":3}"

, errors {

error_code { request_error: UNKNOWN } message: "Queries including ClickView must have a filter limiting the results to one day." }

Hello everyone, tell me how to set the socket timeout parameter for the jdbc driver (connection to the database) on the airbyte side?

I also ran into having to review my facebook app to get around rate limiting. I supplied even videos of setting airbyte with the connector but that didnt help. The review got rejected. Its a joke to use facebook with airbyte when you can’t get the rate limiting removed. Has anyone passed this hurdle successfully? We want to use this connector as an agency to collect data for our clients. But if this doesn’t work then I guess we are stuck with the common solutions like funnel.io, fivetran, or supermetrics.

hello..i have built a connection between mongodb and posgress sql..now i want to ask where the models are created.i couldnot find it?

Hi everyone , I am wondering if it's possible to use multiple fields as a cursor field ?

hey everyone. having an issue whereby airbyte cannot identify any PSQL tables in any schema other than

public

. think it’s similar to https://discuss.airbyte.io/t/postgres-0-4-4-discover-schema-detection-failure/484/9 and have commented there. airbyte version: 0.35.60-alpha postgres connector: 0.4.4 using docker compose, but issue persists with k8s deployment too. here are relevant parts of my logs. does anything look off? struggling to debug. the user account definitely has permissions, as ‘manual’ PSQL queries succeed. nb I’m using a GCP PSQL read replica and CloudSQL proxy to access the DB.

airbyte-worker      | 2022-04-14 11:05:54 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.60-alpha
airbyte-worker      | 2022-04-14 11:05:54 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-postgres:0.4.4 exists...
airbyte-worker      | 2022-04-14 11:05:54 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-postgres:0.4.4 was found locally.
airbyte-worker      | 2022-04-14 11:05:54 INFO i.a.w.p.DockerProcessFactory(create):106 - Creating docker job ID: 4591267a-c9a3-444e-8235-3fcb33a5eccb
airbyte-worker      | 2022-04-14 11:05:54 INFO i.a.w.p.DockerProcessFactory(create):158 - Preparing command: docker run --rm --init -i -w /data/4591267a-c9a3-444e-8235-3fcb33a5eccb/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_CONNECTOR_IMAGE=airbyte/source-postgres:0.4.4 -e WORKER_JOB_ATTEMPT=0 -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.35.60-alpha -e WORKER_JOB_ID=4591267a-c9a3-444e-8235-3fcb33a5eccb airbyte/source-postgres:0.4.4 discover --config source_config.json
airbyte-worker      | 2022-04-14 11:05:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Class path contains multiple SLF4J bindings.
airbyte-worker      | 2022-04-14 11:05:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.16.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
airbyte-worker      | 2022-04-14 11:05:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
airbyte-worker      | 2022-04-14 11:05:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
airbyte-worker      | 2022-04-14 11:05:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 INFO i.a.i.s.p.PostgresSource(main):356 - starting source: class io.airbyte.integrations.source.postgres.PostgresSource
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {discover=null, config=source_config.json}
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: DISCOVER
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-04-14 11:05:55 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:55 INFO i.a.i.b.s.SshTunnel(getInstance):170 - Starting connection with method: NO_TUNNEL
airbyte-worker      | 2022-04-14 11:05:56 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:56 INFO i.a.i.s.j.AbstractJdbcSource(discoverPrimaryKeys):203 - Discover primary keys for tables: []
airbyte-worker      | 2022-04-14 11:05:56 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-14 11:05:56 INFO i.a.i.s.p.PostgresSource(isCdc):253 - using CDC: false
airbyte-worker      | 2022-04-14 11:05:56 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
airbyte-server      | 2022-04-14 11:05:56 INFO i.a.s.RequestLogger(filter):112 - REQ 172.19.0.5 POST 200 /api/v1/sources/discover_schema - {"sourceId":"944a0fac-1bf0-4125-80f9-a0240fbf621f"}

Hello! I did test with Airbyte core - I have created a cloud account, but I don’t see same list of connectors in cloud and core ? For example, BambooHR I don’t see it in Cloud version ? Is there restriction on cloud ?

👋 Ciao, team!

Hi guys, I'm relatively new to Airbyte

I deployed it on an AWS EC2 instance

I set up its source to a mysql database running on an EC2 instance

and a destination to an s3 bucket

I ran the sync

but it failed at some point with this error: destination > tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field location_latitude is expected to be one of these: NULL, DOUBLE. If this is a complex type, check if offending field: location_latitude adheres to schema

Any idea how to fix it?

Hi! Does Airbyte have a connector to Rocket’s U2 database for extraction?

Hello Folks, Getting this error when trying to connect Airbyte with CDC to MySQL RDS. Standard connection works. Airbyte MySQL connector version is 0.5.6

Error: The variable log_bin should be set to ON, but it is : OFF

• On RDS with MySQL 5.7+ we don't actually have that specific parameter for log_bin • So we turned log_bin on simply by enabling automatic backups with a retention period greater than 0 • I believe Airbyte is looking for that specific parameter

log_bin

because it was used on previous versions of mysql and its not finding and therefore its assuming that log_bin is off, even when its actually on. Any help or insights will be greatly appreciated.

Hi there! We just set up an instance on GCP (Compute Engine, n1-standard-2 instance) and created some Gsheets - BigQuery connections. Everything works fine, we're able to set up the connections and sync our data, but sync schedule does not seem to work (see below example, only manual syncs work) - are we missing something here?

Hey there, was testing out Airbyte by ingesting data into DynamoDB and noticed the following schema for all records ingested: • `_airbyte_ab_id`: a uuid assigned by Airbyte to each event that is processed. • `_airbyte_emitted_at`: a timestamp representing when the event was pulled from the data source. • `_airbyte_data`: a json blob representing with the extracted data. • `sync_time`: a tim Is there any way to customize the record schema so that data is not nested within the

__airbyte__data

key?

Hi there! I've been using calculator to check monthly cost. However I couldn't figure out minimum cost measurement unit. Is it correct to count it as 15.00 USD per 1 GB not including rows count?

Hello Everyone, I am working on building a custom image on airbyte. The docker compiles fine. And check for config turns out to be right. However, When I try to add that image to UI, it says that spec jobs failed. Am I missing anything from my custom docker image. Please advise.