Hey guys! Need some help 🙏 I’m developing a custom source connector and have source data structure represented as JSON:

{
   "visitId":"xxx",
   "visitStartTime":"xxx",
   "date":"xxx",
   "geoNetwork":{
      "country":"xxx"
   },
   "hits":[
      {
         "page":{
            "pagePath":"xxx",
            "hostname":"xxx,
            "pageTitle":"xxx"
         }
      }
   ]
}

Corresponding catalog for the schema:

{
  "streams": [
    {
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite",
      "stream": {
        "name": "data",
        "json_schema": {
          "type": "object",
          "properties": {
            "visitId": {
              "type": "number"
            },
            "visitStartTime": {
              "type": "number"
            },
            "date": {
              "type": "string"
            },
            "geoNetwork": {
              "type": "object",
              "properties": {
                "country": {
                  "type": "string"
                }
              }
            },
            "hits": {
              "type": "array",
              "properties": {
                "page": {
                  "type": "object",
                  "properties": {
                    "pagePath": {
                      "type": "string"
                    },
                    "hostname": {
                      "type": "string"
                    },
                    "pageTitle": {
                      "type": "string"
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  ]
}

So effectively,

hits

property is an array where one of properties is an object. I’m struggling with normalization here. Basic normalization works fine for flat nested objects like geoNetwork, but

hits

table not even being created ❓ Do I need to create a custom dbt transform in order to normalize records like these or there’s other way to do this? Thank you!

How big is the airbyte-integrations dir supposed to be? Mine is 5.6G 🤣 My guess is that most of the weight here is the multiple pyenvs.

I’ve been using Slapdash for a few months and it’s been an amazing tool. It’s basically Mac Spotlight, but across all of your apps. Thought it might be interesting to people here

tfw you wake up on a Monday morning and 8 connections are down

Can I rant here to reaffirm your work and what you're up to? 😊

I just tried subscribing to the Airbyte newsletter with a new email. Looks like the form field isn't working - I tried this both on Safari and Chrome on a Macbook.

We are the second result that comes up in google when you search "Failed to exec spawn helper m1". This feels like a form of greatness. But also, we will fix the M1 issues this month!

Really interesting data tool I saw on HN yesterday. It provides an abstraction over any external data source directly in SQL: https://steampipe.io/ demo video:

anyone have a show & tell of insights from the github connector? i know there was a recent demo with superset showcasing this. but possibly a public

airbyte-dbt

package (similar to fivetran dbt package)

Hello, i have a question, i've seen that airbyte always create a destination with name "_airbyte_raw" is there any scenario where is doesn't create it ? and is it possible to delete them after the jobs are run or will this create a problem during the futur syncs ?

for our users, don't hesitate to vote on that poll 🙂 https://www.reddit.com/r/dataengineering/comments/pbaw2f/what_etl_tool_do_you_use/

Hi, Does anyone have any information on the "Webhook connector" mentioned at https://docs.airbyte.io/project-overview/roadmap - either on its progress/ETA or perhaps what it might provide? Thanks

who made all the Airbyte/Octavia emoji? they are great!

Poll: What’s your opinion on DataOps? 1️⃣ Don’t care, most people are not ready 2️⃣ Only needed at large companies over 600 employees 3️⃣ I wish I could do it, but don’t have the skills 4️⃣ I do it, but set up is difficult 5️⃣ Whats DataOps?

anyone here make a transition from API development to data engineering? I am going to be doing that and wanted to hear about other people backgrounds, unless there is a more preferable channel for that

The Airbyte team is a big fan of Among Us, we play it together at least once a week. Great to see that we will be able to do it in real life :) https://www.theverge.com/tldr/2021/9/15/22676173/among-us-official-costume-impostor-crewmate

Screen Shot 2021-09-16 at 9.48.35 AM.png

I just found out that Airbyte owns the .com domain too. Strange, why do you prefer the .io domain @Michel?

Hi everyone, I had a question ! This may have been already asked but ... How does Airbyte scale data migration ? Is there an in-house developed MPP system being used to distribute and manage a HUGE ingestion job across multiple workers ? Regards, Elaaf

Hi everyone, I am planning to use Airbyte in our team for replication. We are currently using Redshift and are evaluating Snowflake as a potential warehouse alternative. Does anyone know if it's easier to move the ingestion task destination from redshift to snowflake (assuming the existing data is moved)?

is this an addon which keeps your old files without deleting them on a free plan?

Good night, I have airbyte on AWS and working perfectly, has anyone ever used auth0 to put a user authentication layer? or some other user authentication tool? (I know that it is possible to release some ip’s in the airbyte vpc but I would like to do it with authentication, so I would not always need to be putting the ip)

👀 https://www.independent.co.uk/life-style/gadgets-and-tech/facebook-down-instagram-whatsapp-today-b1932059.html

Hello everyone 🙂 I am curious to know when you plan to integrate Prefect ?

Hello community, I got a question about the mysql integration and log based replication. Does the integration handle changing log files? When the log file reaches its size limit, mysql creates a new log file with a higher index (ex: bin-log.00001 -> bin-log.00002). The stitch integration does not handle this...

Hi all, I’m curious, what’s the recommended way to run airbyte on kubernetes? I found a helm chart but I also saw some kustomize related files here and I was wondering which one should I choose.

Hi people, I am using airbyteairbyte along with other services to play with the data. I was curious if I can use airbyte-db (postgres) to store data specific to other services ? 🤔 just create another database for other services to store there data. Will this affect airbyte in any way?

Hi all, I’m trying to check with the rest api if a sync finished with any status however I couldn’t find the corresponding api call. Is there a way to achieve this with the API?

Cool mini-game to find your favourite code font for IDEs: https://www.codingfont.com/