Hello Team! Airbyte OSS user here. Anyone having problem ingesting Salesforce's Case to Snowflake? 🙂

Hi Team, I’m setting up the ingestion with MySQL as the source but it’s failing with the Error Msg -

The connection tests failed. Could not connect with provided configuration. Error: HikariPool-1 - Connection is not available, request timed out after 60007ms

Attaching the screenshot of the same. I have tried this with both Standard & CDC Replication Method. I have checked the connectivity with MySQL host from the server where Airbyte is hosted and it’s working so not really sure about this issue. Can someone please help here?

Does anyone have any insights into the docker desktop licensing required in a scenario where you'd be using airbyte OSS in a commercial setting, ie, at your company? Have a single node server with airbyte in docker.

Hi all, new to Airbyte and I have built the container image locally but failed to create a connection to Hubspot source.. it seems to just hang..I also tried to connect to MySQL database and the same is happening..any ideas what the issue may be?

Hi. Can I have two postgres sources sending data to a same destination? Is it a good practice? I am trying to split our postgres data ingestions in groups. Thank you.

Hi, is there a channel specifically targeted to discussions of connector development/issues? My company, Zeenk, is using airbyte to pull a variety of marketing data for use in our platform, with good results so far. My colleague @Chen Lin is looking into replacing our custom Facebook Ads Api implementation with Airbyte, and found some issues in the Facebook Ads source connector that we would like to work on - but wanted some discussion about them first to make sure we follow an approach that will fit in.

hey everyone, i'm trying to sync my google analytics data to a big query instance. It's been running the initial sync for over two hours and its only synced ~5gb of data so far. Is this normal? what can i do to speed this up?

hi! I get errors syncing data using stripe to snowflake (although I assume its a global cluster issue). this started to happen after cloning my k8s cluster to a new eks cluster (along with upgrading from

0.35.64-alpha

to

0.39.36-alpha

the clone I did was to just to basically to take a postgres dump -> redeploy the cluster on a new eks cluster -> postgres restore all configurations kept the same. the error I get is:

Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:138) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:158) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:284) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	... 14 more
Caused by: software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 1702D3276AA70C1C, Extended Request ID: null)
	at io.airbyte.workers.storage.S3DocumentStoreClient.read(S3DocumentStoreClient.java:75) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocument(AsyncOrchestratorPodProcess.java:216) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.process.AsyncOrchestratorPodProcess.checkStatus(AsyncOrchestratorPodProcess.java:220) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocStoreStatus(AsyncOrchestratorPodProcess.java:229) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:137) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:315) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]

Hi all, I'm trying to specify the namespace (schema) in my MSSQL source. The documentation says it should be possible, and using namespaces seem to have been merged in PR https://github.com/airbytehq/airbyte/pull/2990. However, I don't see how I'm able to specify the namespace through the UI. Am I missing something? Airbyte version 0.39.37-alpha MSSQL source version 0.4.9

Hi guys... does anyone have a good tutorial on how to connect "Amazon Seller Partner" to Airbyte ? 🙏 I already tried to follow the documentation steps... but it is not working 😞

Hello! I have an S3 bucket full of JSON files. It seems like the S3 source only supports CSV, Parquet and Avro. There's also the File source, which seems to support JSON but has to be configured on a per-file basis. Is there a different source/connector I should be using? Or a way to pattern match the File connector to match all files in an S3 bucket?

On Airbyte Cloud, I want to create a Source for an internal Postgres db. For security reasons, is there an IP address that I can add to an allowlist from our side, so that we’re not exposing our Postgres server to the entire internet?

Hi Guys, I am wondering if anyone had experience saving data to an NFS Storage, is there a connector that can work with NFS to save the data as JSON?

what’s the difference between

https://docs.airbyte.com/integrations/sources/mongodb-v2

and

https://docs.airbyte.com/integrations/sources/mongodb/

Wrote a schema json when building my own http api connector. But all data is dropped in a single row. Any ideas what I might be doing wrong?

Has anyone tried extracting postgres

information_schema

schema using Airbyte? Does it work?

Is this source mongodb connector here using the oplog for incremental capture? I don’t see any reference to it in the documentation, but I assume it is.

Hello!! Currently I have a Airbyte deployed using Helm on a Kubernetes, which has a lot of free resources, but while trying to extract a small table(less than 10000 lines) the Sync job keeps failing with

137

exit code, how can I scale the Airbyte Sync job? Thanks in advance!

Hi everybody! I'm trying to sync MySQL to Redshift and Airbyte hangs. In logs, there are messages saying

Thread starvation or clock leap detected

. I run it on the EC2 t2.medium instance. Any ideas on what to do? Do I need to switch to a bigger instance?

Greeting: I’m a newbie in the Airbyte community. my question of the best practice to handle local file uploading, for example, CSV, and JSON files to my configured destination in the workspace. I searched in the documentation and I found 2 connectors related to files one for the source and the other one is destination. what if the user wants to upload my local file and I want to make it a connection source. how can I handle this streaming data? is the only way to achieve this creating http api connector ?! if someone went through similar problems it would be awesome to share the troubleshooting

Hello there, I'm new to using Airbyte. I don't know if this is the right channel for this. Anyone is syncing data from Stripe? I'm having a bad request on the subscriptions endpoint:

requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <https://api.stripe.com/v1/subscriptions?limit=100&created%5Bgte%5D=1467331200&status=all> 

{
  "error": {
    "message": "Invalid status: must be one of active, past_due, unpaid, incomplete, incomplete_expired, trialing, or paused",
    "param": "status",
    "type": "invalid_request_error"
  }
}

Any chance this error was discovered previously and about to be fixed?

Hello 👋 I’m sending this message to help you identify if this channel is the best place to post your question. Airbyte has a few channels to open discussion about data topics (architecture, ingestion, quality, etc). In these channels you may ask general questions related to the particular topic. If you’re having problem deploying or running a connection in Airbyte this is not the topic. We recommend to you open a Discourse Topic where our support team will help you troubleshooting your issue.

Hi do you have a connector for the yahoo dsp reporting api?.

Hi all! Anybody else having issues ingesting costs data from Google Ads, using Airbyte? We are getting negative values, which do not match what we see in the actual Google Ads platform. Has anyone else seen this? This is specifically from the metrics.cost_micros field.

Hi all, I am a bit curious to understand how CDC works currently with Airbyte. Please provide inputs or route to some resources I can go through. Thanks.

Hi all, is it possible for CDC in Postgres to miss some record updates? I have an

orders

table with a

state

field that changes every time the order moves state and a

cdc_updated_at

field used as a cursor to detect changes in the table. I’ve identified an order that has a different

state

in my origin DB and my destination data warehouse. The order’s state was updated one month ago but somehow it wasn’t picked up by Airbyte. I’ve compared with other orders and all of the ones I checked are being updated properly It’s a question difficult to answer because I can’t find any logs on the job in order to identify the issue but just wondering if someone has come across with something similar!

Hello 👋 I’m sending this message to help you identify if this channel is the best place to post your question. Airbyte has a few channels to open discussion about data topics (architecture, ingestion, quality, etc). In these channels you may ask general questions related to the particular topic. If you’re having problem deploying or running a connection in Airbyte this is not the topic. We recommend to you open a Discourse Topic where our support team will help you troubleshooting your issue.

whenever, I try to to pull data from facebook I get

2022-08-02 06:57:59 source > Utilization is too high (114)%, pausing for 5 minutes

what is the settings that can help me bring data in batches

Hi, I receive and error when trying to pull data from Exchanges Rates Api. I requested the free access_key, and tested with a new source and get: ‘You have not supplied a valid API Access Key.’ Airbbyte version 0.35.28-alpha. Any idea on what’s going on? I am 100% sure that api key is the correct as is provided through APILayer.com

2022-07-05 10:15:53 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/5bf8952f-6242-4183-9bc2-f3486faf4a68/0/logs.log
2022-07-05 10:15:53 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.28-alpha
2022-07-05 10:15:53 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-exchange-rates:0.2.5 exists...
2022-07-05 10:15:53 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-exchange-rates:0.2.5 was found locally.
2022-07-05 10:15:53 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/5bf8952f-6242-4183-9bc2-f3486faf4a68/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/source-exchange-rates:0.2.5 check --config source_config.json
2022-07-05 10:15:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - Ping response code: 401
2022-07-05 10:15:54 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Check failed
2022-07-05 10:15:54 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...

hi.. can someone guide me, how to increase fetch size limit of the source ?