Hi All,

I'm looking to connect with someone who has a (succesfull) Amazon seller partner connection on OSS

hey, is there any API where I can get the schema information (at least the table names) generated by airbyte during

Normalize Tabular data

step? I can retrieve the stream table names, but not normalized tables created by the built in basic normalization

Simply cannot get airbyte-webapp to work 100% behind NGINX reverse proxy. Has anyone successfully done this with Windows and local deployment in docker with NGINX in wsl on same localhost? airbyte-webapp on localhost:8000 nginx on localhost listening on localhost:200 proxy_pass to http://localhost:8000/; airbyte-webapp successfully receiving the get requests when hitting localhost:200 manifest.json, random static/js/chunks timing out basically instantly on the airbyte-webapp side nginx timeouts are set to 1200 (proxy_connect_timeout, proxy_send_timeout, proxy_read_timeout, fastcgi_send_timeout, fast_cgi_read_timeout, send_timeout DevTools failed to load source map: Could not load content for http://localhost:200/static/css/224.e564b584.chunk.css.map: Load canceled due to load timeout ChunkLoadError: Loading chunk 107 failed. (timeout: http://localhost:200/static/js/107.9e468796.chunk.js) 2022/08/31 122844 [error] 2195#2195: *4870 upstream timed out (110: Connection timed out) while reading upstream, client: localhost, server: localhost, request: "GET /static/css/224.e564b584.chunk.css.map HTTP/1.1", upstream: "http://127.0.0.1:8000/static/css/224.e564b584.chunk.css.map", host: "localhost:200"

simple question: when debugging locally a python connector how does one set the Log Level to DEBUG? ’m not inside a container. simply testing within PyCharm so that

logger.debug(f"Processing stream slices ...

will appear in the logs for instance nevermind 😅 it’s as simple as adding the

--debug

flag to the python command

Hello Airbyte Team! Is there a connector for HotJar? 🔥

Is there like a quick pros and cons for the two bigquery connector modes? • bigquery with basic normalization • bigquery(denormalized) Not sure which one to use. there is plan to run separate dbt transformations on the data after airbyte extracts them to bigquery. any insights is appreciated!

Hello, I am quite new to airbyte and analytics engineering in general. My marketing team setup our google analytics using the GA4 and I have been waiting to update. It is my understanding that as of v0.40.1 the required view_id was fixed. I just updated our version to v0.40.3 but I still see the view ID as a required field. I have also updated the google analytics version to the latest as well. Is there something I am still doing wrong? I apologize ahead of time if this is not the proper place for this question.

Hello everybody. Does anyone know what the current state is for the goland SDK? A few months ago I was excited with the announcement but then I had to pause my project with Airbyte. Any connectors with this SDK? good reviews so far?

Hello everybody 👋 I am trying to generate a BigQuery destination using octavia-cli. Everything works fine until testing the connection from Airbyte UI :

com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 1 path $

Are there any guidelines on where to store the required environment variables ? Until now, I tried : • exporting them before

octavia apply

• rebuilding a custom airbyte/worker docker image, with credentials stored in env variables I may precise that the destination worked as expected when created from the UI. ------------- EDIT : always better to read the docs entirely 😉 • Env vars need to be set either in your

.octavia

file, or in your custom octavia docker image (I chose this option)

Hi guys 🙂 I'm quite new to AirBytes and I have a simple question for you (hope so) about Jira Connector: Does this connector works for both Jira Cloud and On Premise version? I didn't find anything on the documentation about that. Thanks in advance, Stefano.

Hi everyone, I'm trying to understand how to handle schema changes in sources that Airbyte reads as json and then flattens. I'd like to extract data from Salesforce and use Airbyte's flattening features to create the schemas of the destination tables based on the JSON response from Salesforce. If a new column is added to a source table being extracted by Airbyte, is Airbyte capable of automatically detect this new column and add it to the destination table?

Hey everyone, I'm trying to create a TypeScript destination and I noticed there aren't any docs for this. What's the best way to get started? I'm intending to create a pipeline from Airtable to a custom REST API destination.

Hello Team, I am super new to Airbyte I can t find a simple step-by-step guide to follow for setting up the open source. Can anyone point me on the right direction please ❓ Thanks

👋

Hey, friends! I'm using Airbyte at my work for 6 months and have developed number of custom connectors: • For Google Play app reviews; • For AppStore RSS and AMP API app reviews; • For Trustpilot app reviews. I'd like to contribute them to Airbyte, but before I'd like to understand: will they be helpful for anyone? If someone from Airbyte core will answer me, it'll be really cool

Maybe a silly question: how do I send data from a source (Airtable) to a destination (Postgres) while preserving the relationships? I have some relationships between Airtable tables that don’t move over to Postgres when I run the pipeline. Is this more a question for the Airtable team?

Hi team, I’m using

airbyte 0.40.3

with a

MySQL

source of version

0.6.5

, and my source table has a primary key and an

updated_at

field which can be set as a cursor field. but I can’t choose

incremental sync Deduplicate history

sync mode. according the doc, it seems that I am able to use the

deduplicate history

sync mode, so I’m wondering if there are something I missed?

Hey team! 🙂 I have a quick question regarding batch size in Airbyte. I deployed a custom HTTP source connector which queries large amounts of data. With the apparently hard-coded batch_size of 10K records, my machine (32GB RAM) runs out of memory. How can I adjust the batch size? I saw that this was recently added for JDBC-based connectors. Is an equivalent available for the Python cdk? Thank you!

Morning all wondering if there is a way of non-UI based configuration for source, destination and replications? Can this be done in one or several config files placed, like on the airbyte server container

Hey, I have a Facebook to BigQuery connection, that worked very well for over a week. Now I receive the following error message: FacebookAPIException(‘Error: 190, Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.’) The password has NOT been changed, so it must have been changed by Facebook. I managed to get a response from the Meta Team they need a request submitted in their Graph API explorer. Did anyone already go through this? What exactly do they want? I will ask them as well, but their first respond took over half a day and I would like to fix this ASAP. Thank you

Hi, I am trying to create my first destination and getting the below error. Can anyone explain what exactly this means or how to resolve it? thanks!

Hi team, I ran into another issue with one of my custom HTTP connectors and would be very thankful if you could help. When running the source-connector from the console using the following command, everything works fine:

docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/<CUSTOM_SOURCE>:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json

However, the Airbyte GUI throws the following error message and the sync fails (I tried it with both S3 and local json destinations):

airbyte-worker      | 2022-09-06 12:46:11 ERROR i.a.w.g.DefaultReplicationWorker(run):184 - Sync worker failed.
airbyte-worker      | java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "java.net.URI.toString()" because "currentUri" is null
airbyte-worker      | 	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
airbyte-worker      | 	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
airbyte-worker      | 	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:177) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:65) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at java.lang.Thread.run(Thread.java:1589) [?:?]
airbyte-worker      | 	Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.
airbyte-worker      | 		at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:141) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:65) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 		at java.lang.Thread.run(Thread.java:1589) [?:?]
airbyte-worker      | Caused by: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "java.net.URI.toString()" because "currentUri" is null
airbyte-worker      | 	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:372) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
airbyte-worker      | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
airbyte-worker      | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
airbyte-worker      | 	... 1 more
airbyte-worker      | Caused by: java.lang.NullPointerException: Cannot invoke "java.net.URI.toString()" because "currentUri" is null
airbyte-worker      | 	at com.networknt.schema.JsonSchema.combineCurrentUriWithIds(JsonSchema.java:90) ~[json-schema-validator-1.0.42.jar:?]
airbyte-worker      | 	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:72) ~[json-schema-validator-1.0.42.jar:?]
airbyte-worker      | 	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:58) ~[json-schema-validator-1.0.42.jar:?]
airbyte-worker      | 	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:53) ~[json-schema-validator-1.0.42.jar:?]
airbyte-worker      | 	at com.networknt.schema.JsonSchemaFactory.newJsonSchema(JsonSchemaFactory.java:254) ~[json-schema-validator-1.0.42.jar:?]
airbyte-worker      | 	at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:362) ~[json-schema-validator-1.0.42.jar:?]
airbyte-worker      | 	at io.airbyte.validation.json.JsonSchemaValidator.validateInternal(JsonSchemaValidator.java:63) ~[io.airbyte-airbyte-json-validation-0.40.1.jar:?]
airbyte-worker      | 	at io.airbyte.validation.json.JsonSchemaValidator.ensure(JsonSchemaValidator.java:78) ~[io.airbyte-airbyte-json-validation-0.40.1.jar:?]
airbyte-worker      | 	at io.airbyte.workers.RecordSchemaValidator.validateSchema(RecordSchemaValidator.java:51) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at io.airbyte.workers.general.DefaultReplicationWorker.validateSchema(DefaultReplicationWorker.java:393) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:321) ~[io.airbyte-airbyte-workers-0.40.1.jar:?]
airbyte-worker      | 	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
airbyte-worker      | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
airbyte-worker      | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
airbyte-worker      | 	... 1 more

Hi, I am building a custom source connector with multiple streams and had a few questions: 1. Does Airbyte automatically sync all streams? 2. Is there a way to configure which stream is sync'd during a given execution? Thanks in advance for the help!

What is the best way to set up multitenancy on open-source Airbyte? By multitenancy I mean several groups will have access to Airbyte to create their own ingest pipelines but shouldn’t be able to see sources/destinations/connections/logs other than their own. Do workspaces help with this? I don’t see workspaces mentioned much in the documentation (although I see them in the API reference.)

Hello, I am wondering if there are any good resources to learn about data transformation of airbyte data using s3 as a destination. I understand that airbyte itself doesn't allow transformation to the s3 destination, but would love to eventually use that as a destination instead of redshift and normalize in dbt.

I’m currently trying to get my source connector PR merged and I have one question about this process. Should I build and upload my connector image to dockerhub?

I would like to normalize the following response JSON from external API in my python source connector to something simpler. for example i want to create anther JSON where "avgCPA":{"amount": "0.974", "currency": "USD"} to be converted to avgCPA_amount, avgCPA_currency. Is there a pluggable way to do this before sending JSON to the postgres destination.

{
  "type": "RECORD",
  "record": {
    "stream": "campaigns",
    "data": {
      "campaigns_report": [
        [
          {
            "other": false,
            "total": {
              "impressions": 1254,
              "taps": 32,
              "installs": 9,
              "newDownloads": 9,
              "redownloads": 0,
              "latOnInstalls": 0,
              "latOffInstalls": 0,
              "ttr": 0.0255,
              "avgCPA": {
                "amount": "0.974",
                "currency": "USD"
              },
              "avgCPT": {
                "amount": "0.274",
                "currency": "USD"
              },
              "avgCPM": {
                "amount": "6.9907",
                "currency": "USD"
              },
              "localSpend": {
                "amount": "8.7664",
                "currency": "USD"
              },
              "conversionRate": 0.2813
            },
            "metadata": {
              "campaignId": 111111,
              "campaignName": "Campaign1",
              "deleted": false,
              "campaignStatus": "ENABLED",
              "app": {
                "appName": "App1",
                "adamId": 1111
              },
              "servingStatus": "RUNNING",
              "servingStateReasons": null,
              "countriesOrRegions": [
                "US"
              ],
              "modificationTime": "2022-08-29T18:23:58.284",
              "totalBudget": null,
              "dailyBudget": {
                "amount": "5",
                "currency": "USD"
              },
              "displayStatus": "RUNNING",
              "supplySources": [
                "APPSTORE_SEARCH_RESULTS"
              ],
              "adChannelType": "SEARCH",
              "orgId": 1,
              "countryOrRegionServingStateReasons": {},
              "billingEvent": "TAPS",
              "countryOrRegion": "US"
            }
          },
          {
            "other": false,
            "total": {
              "impressions": 121,
              "taps": 6,
              "installs": 2,
              "newDownloads": 2,
              "redownloads": 0,
              "latOnInstalls": 0,
              "latOffInstalls": 0,
              "ttr": 0.0496,
              "avgCPA": {
                "amount": "2.265",
                "currency": "USD"
              },
              "avgCPT": {
                "amount": "0.755",
                "currency": "USD"
              },
              "avgCPM": {
                "amount": "37.438",
                "currency": "USD"
              },
              "localSpend": {
                "amount": "4.53",
                "currency": "USD"
              },
              "conversionRate": 0.3333
            },
            "metadata": {
              "campaignId": 222222,
              "campaignName": "Campaign2",
              "deleted": false,
              "campaignStatus": "ENABLED",
              "app": {
                "appName": "App2",
                "adamId": 2222
              },
              "servingStatus": "RUNNING",
              "servingStateReasons": null,
              "countriesOrRegions": [
                "US"
              ],
              "modificationTime": "2022-08-22T17:05:28.945",
              "totalBudget": null,
              "dailyBudget": {
                "amount": "15",
                "currency": "USD"
              },
              "displayStatus": "RUNNING",
              "supplySources": [
                "APPSTORE_SEARCH_RESULTS"
              ],
              "adChannelType": "SEARCH",
              "orgId": 2,
              "countryOrRegionServingStateReasons": {},
              "billingEvent": "TAPS",
              "countryOrRegion": "US"
            }
          }
        ],
        [
          "other",
          "total"
        ]
      ]
    },
    "emitted_at": 1662110857000
  }
}

Hello everyone. How do i process data from FTP match with SQL data format?

It's just has source and destination , i don't where can i process data