Do you have any idea when S3 Destination connector will support JSON/XML/EXCEL?

Question on setting up airbyte on cloud composer. Should I be installing the docker images on a separate compute instance and creating a connection to that instance on airflow to call the container?

Hi everyone :) My name is Tom Griffin - I’m experimenting with Airbyte to build out a vaccination warehouse for a Albany County in New York. One of the data sources is the state’s immunization registry. Each day they cut a csv of the prior day’s vaccination events. We pull that file, ingest it, etc... We ended up writing a small python script that syncs their SFTP directory with a local directory on our end (based on filenames). We went this route because there were days where they dropped more than one file and we didn’t want to risk losing anything. I experimented with the SFTP connector and was able to download specific files I defined as part of the source configuration. I was unable to match all files in the directory, like /directory/filename2021*.csv. What would be the preferred method to mimic what I am doing now in terms of the directors synchronization and then only processing the new files (with new defined as those that I didn’t have before the job ran)? For example, is there a way that I could trigger a script to run beforehand that would stage the data locally? Any ideas would be appreciated :)

Can i import these project's in Eclipse as gradle project's?

Hello everyone ! I'm trying to configure my first connection between a MSSQL and a PostGres database. Is there a way to use the windows authentification with the MSSQL connector ? Thanks !

Hi all! Just getting started with Airbyte. I'm a current Fivetran customer and trying to see if Airbyte will solve some of our needs. The first thing I'm trying to do is clone our S3 file data sources (CSV files) to a test postgres destination and running into some issues: 1. Any null string column is written as

NaN

2. the column names are written in title case (eg

Job Type

) and it would be great if it could be cleaned as snake_case 3. column type mappings - seems like timestamps are kept as strings. Any control over that?

Hi all, • Is there any way that I can managed the configuration files with a git repository. ◦ Expecting that when I change the configurations files content and commit it to the git, then I can sync it into Airbyte container and it will works, no need zip or containers restart, • Also, where are those file stored? Maybe we could find some way to adapt if it feasible. For our use case, managing and version control setting is one of the crucial points, some of our team members are prefer some other tools due to its yml configuration style and CLI, but i don’t think Airbyte can’t also do the same thing, so i want to find a way. Thanks a lot

Looking for a little more information around the Oracle source connector. Does Airbyte support Oracle container databases?

Hey, quoting https://github.com/airbytehq/airbyte/blob/master/docs/deploying-airbyte/on-aws-ec2.md :

If you want to use different ports you will need to modify 

API_URL

 in your 

.env

 file and restart Airbyte.

What is the exact syntax to specify different port in API_URL in

.env

file?

Hi all and thanks in advance. Im trying to seek Airbyte quickstart and just with that I'm having some issues. I share some screenshots inside threat.

Hi All, Testing out Airbyte as a migration tool. I can see that Airbyte supports MSSQL, but I cant seem to find if it support Windows Authentication? Thanks in advance. Cheers,

Hi! which benefits comes with specifying a primary key in a stream?

Looking on opinions, critique, and also some help on this practice. I am moving sources into big query. I want all sources to end up in it's own schema. • When using big query as a destination I can't seem to override the default schema. I have to edit the default schema before each sync manually. Am I missing the place within the sources to do this? • What are people's opinion on a schema per source and then creating additional schemas for the transformed/business friendly data. I assume this practice is fine as long as it works for the business? These may be amateur questions but my background isn't old school BI engineer and actually come more from a data analyst/scientist background. For my current role for now it seems I will be doing the whole end to end from the engineering to the 'actionable insights' (or whatever buzz term you want to use)

I am trying the basic replication sample, but instead of Postgres, I have configured MS SQL Server as both Source and Destination. Source DB

airbytesrc

has a single table EMPLOYEES with column (ID, FN, LN, STATUS, MODIFIEDON) in dbo schema, all String type including MODIFIEDON. ID is the PRIMARY key. The destination database is

airbytedest

. I have specified 'airtbyte' as the 'Default' schema. Sync setting is "Full Refresh" append. Currently manual sync is being used and it succeeds to replicate the rows. However in destination, the data is in a table with name

_airbyte_raw_EMPLOYEES

with only three columns

_airbyte_ab_id, _airbyte_data, _airbyte_emitted_at

and the row data from source is in the column

_airbyte_data

in Json format. My question is - after sync, should't the data be in the destination DB with table airbyte.EMPLOYEES with all source column, instead of in the "raw" table in Json format? This is my first attempt to use Airbyte, so I might have misunderstood this.

I have a connection set up (MySQL --> Snowflake), which loads 1 table. I would like to add another table so I go to destination settings

Select the data you want to sync

but i would like to add this other table so I click update schema to see the additional tables

To save the new schema, click on Save changes. Note that it will delete all the data in your destination and start syncing from scratch.

I would like to add this other table without deleing all data in destination and sync from scratch. I appreciate that it thinks the overall schema is changing but its just adding another table Is the only way to do this, to create a duplicate source?

Question: Is there a way to trigger a ELT run for a connection externally, rather than, or in addition to, a set schedule? I see the /v1/connections/sync API call, but that requires a connectionId, which you can't get with the /v1/connections/list unless you know a workspaceId, and there is a /v1/workspaces/get_by_slug, but there is no discussion of workspaces in the docs I can see on a quick scan, nor is there anything in the UI for workspaces. Also, no info on how to get/set an API key for the API calls.

Hi all. Here a very dumb question. I'm having some issue trying to connect with a simple google spreadsheet resource. Details inside thread, thank You.

Hi all. I’m searching the docs, but having a hard time figuring this out. Can we write a standalone command-line app using Airbyte libraries and connectors? That is, we want to pull some CSV or JSON into our database with a simple command-line tool. We also want to be able to run in a typical configurable flow with a GUI, but the command line config is a requirement for us. Does Airbyte have underlying libraries and .jar files and such to use this way?

Hi all, I am trying to use an opensource ETL, which helps me to extract transform and load data from multiple postgres db's to single postgres db, can I do it using Airbyt. I was able to extract and load but how can I perform Transformation of data, I can't seen any option for transformation in Airbyte, please help with this I am struck at this point

Hi! First of all thank you for developing such a great product! Already using it for relational sources and it works flawlessly. I am now trying to load data to BigQuery using Mailchimp source. Connector setup works with no problem, however the sync process always ends up with 0 records. I hope i am not missing something trivial here, but I cannot find any hint on why no record is being read.

Having some trouble getting started on my apple silicon mac - using

docker compose

I can get the app up and running (though it usually hangs the first time and I have to cancel the docker compose and restart it). The problem I’m running into is either a 500 error or a timeout when trying to configure a source. The endpoint in question is

<http://localhost:8000/api/v1/source_definition_specifications/get>

. I’ve tried

docker compose down -v

to start from scratch, but the same issue reoccurs. (also side-note, the support link shown in the screenshot 404s)

Hi everyone, thanks for developing the wonderful product. Is there any alert setup if connection sync failed and other alerts? I searched through the github issue and the docs on the main site but I can't seem to find anything regarding it. Thanks in advance!

Sorry, one more question. Is it possible to use private singer tap for airbyte (given that we convert it) or does it have to be a public singer tap?

What is the process for using the airbyte flows on our platform please share the information related this.

hello airbyte team! quick question: is powered by airbyte an OEM offering, or a hosted self-serve option for airbyte?

Anyone have a shell script to upgrade Airbyte?

Is there an AirByte image available in Docker Hub?

hi can I use AirByte with dbt cloud?

Trying out the custom dbt transformation and wondering how the final schema is generated (

Namespace Custom Format

vs dbt project

+schema

) e.g.

Namespace Custom Format = "exchange_rates_api"

dbt_projects.yml
...
models:
  airbyte_dbt_exchangeratesapi:
      exchange_rates:
        +materialized: table
        +schema: exchange_rates_api

however, when running this sync. the output schema is

airbyte_exchange_rates_api

my question is. where is the prefix

airbyte

being created?

Hi, I am trying to get a developer token from Google Ads that I can use with Airbyte connection. Google is asking me to provide design docs. They want an actual document rather than a link to the airbyte docs. Can anyone help?