Hey All, We'd prefer not to store our source and destination passwords within the Airbyte DB as we'd need our to be able to roll passwords without anyone needing to handle the passwords or having an application which needs to update the DB . Do you have an existing way to pull passwords at run time from a service like AWS secrets manager?

Hi! What is the official Airbyte position on providing Sources as opposed to Targets? Totally un-scientifically, it feels like Sources are the priority with fewer Targets being released? https://airbytehq.slack.com/archives/C01A4CAP81L/p1627683548002800 Thanks

Hello....can you advise if

runtime column propagation

is in the works and the same for new tables?

hopefully this is the right channel to post in. deploying airbyte on an EC2 instance. whats the easiest way to push docker logs to cloudwatch

how can I schedule connections? currently I have 8 FB Ads connection (8 ad account) that run hourly. And if I dont mistake sometimes they ran at the same time and eat all CPU and make connection cancelled all. How can I schedule them like CRON?

Has anyone tried passing a --var to a dbt transform? I tried

run --var '{"tenant": "xzy"}'

and the command line is getting generated as:

2021-08-03 21:39:07 INFO () LineGobbler(voidCall):85 - Running: dbt run -t prod --var {"tenant": "xyz"} --profiles-dir=/data/588/1/transform --project-dir=/data/588/1/transform/git_repo

Any help on the escaping in the syntax would be appreciated.

Hi, I am trying to set up a google ads connector - however in order to get the necessary token I need to provide documentation for Airbyte. Which documentation should I use for that?

I’m using MongoDB as source. We recently added a new collection to our db. The only aption I see on Airbyte is to completely wipe past data from our destination and start syncing all collections in the db from scratch. Is there another way?

Hi, do you have an ETA or plan for

Apple Search Ads

Source? https://github.com/airbytehq/airbyte/issues/2982

Hello guys! I'm using MWAA on a project and I was wondering, is there a way to integrate Airbyte into my solution without using external compute, like EC2, or ECS?

I’m not sure if this is the right place to post this, but when setting up a connection to move data from a PostgreSQL DB source with over 200 tables, it distorts the UI and makes other form fields inaccessible.

Has anyone looked at a source for UKG ?

Is there a way to trigger a job run via the API? I see things to list, cancel get information about them, but nothing that looks like a way to kick one off.

Hi Everyone, I want to set up Airbyte installation on an AWS EC2 instance. But I don't want to expose the airbyte dashboard (UI) port on the instance as this will mean anyone with the public IP address and the port number can access it. I also read from the documentation that authentication is not yet supported. How can I place restrictions on this port?

Is there a way to prevent that?

Greetings everyone. Is there an API wrapper for airbyte that will allow me to store sources/destinations/jobs as code? I realize I could build one but I was hoping someone has done that already!

I’m extracting documents from MongoDb into Postgres. I’m having problems accessing the nested objects in Postgres. I’m not sure if the fix is in the MongoDB source or my by increasing my skills in Postgres. Document in MongoDB.

{
    "_id": {
      "$oid": "60a413694f65a9002bdc4ff3"
    },
    "__v": 0,
    "activeStatus": {
      "_id": {
        "$oid": "60ad4c2262070d4a8410e537"
      },
      "statusId": 14,
      "taskId": 8,
      "createdOn": {
        "$date": "2021-05-25T19:12:34.045Z"
      },
  }

Selecting the data from the raw table.

select 
	b._airbyte_data::json
from 
	postgres.load._airbyte_raw_b b 
where
	b._airbyte_data::json ->> '_id' = '60a413694f65a9002bdc4ff3'

{
  "__v": 0,
  "_id": "60a413694f65a9002bdc4ff3",
  "activeStatus": "{\"statusId\"=>14, \"taskId\"=>8, \"createdOn\"=>2021-05-25 19:12:34.045 UTC}"
}

The problem I’m having is that the activeStatus column is a string not JSON and I don’t see how to easily parse it as the string is not formatted as JSON. I was expecting to find something more like this.

{
  "__v": 0,
  "_id": "60a413694f65a9002bdc4ff3",
  "activeStatus": "{\"statusId\":14, \"taskId\":8, \"createdOn\":\"2021-05-25 19:12:34.045 UTC\"}"
}

Were the activeStatus column was valid JSON and I could easily parse it with to_json() or that it would have been parsed with the ::json cast.

hi folks! I think something is broken with the smartsheet connector. Even when I sync raw json rows into bigquery, the column headers are not correct. I called the API directly with a quick python script, and the column ids in the cells do correctly line up to the column names correctly in the raw API response.

I'm thinking instead of attempting the side-car to deploy the

cloud-proxy

as it's own service perhaps?

Hi team, is there anything potentially in the works for a more granular customization of sync frequency? I’m interested in more frequent syncs during work hours and less frequent after work hours (and weekends)

Hi all, I'm having difficulties accessing the Facebook Marketing API, as doing that requires a dev account, and a Facebook-verified application a la a SaaS. We are facing difficulties with Facebook in getting Marketing API access for Airbyte. Specifically, I'm having difficulties with FB on the "Request a rate limit increase from Facebook" part. How have you and your users dealt with this? Can there be an alternative authentication method? Like using a Facebook sign in popup? I've used this method to authenticate installing Facebook Pixels on some apps, for example.

Any updates on Clickhouse as a destination? Last I heard, you were bandwidth-limited on this. I note that of all the destinations airbyte currently supports, all OLAP destinations are cloud services. For the open source destinations we only have MySQL and PostgresQL, both OLTP (excluding Meilisearch and Kafka). Having an open source OLAP destination, such as Clickhouse, will be a game changer! :)

Hi, is it supported to connect multiple sources (i.e. multiple instances of Facebook Marketing, each pointing to a different account) to the same destination (i.e. same Redshift tables)? would something conflict? specifically, I am interested to sync ads_insigths incrementally (and possibly deduped if a PK exists) (generated Redshift tables have a minimum size of 47 GB due to all the columns and sub-tables)

is there an updated ETA on the conversion of the existing Mongo connector to Java?

Hi 👋 I remember from the July community call that SSL for DB connectors was a priority for last month. Is there an update around this?

is the generic HTTP source gone?

Is this the correct channel for general error reporting?

Where can I find the changelog to each new connector update?

We are having some difficulties getting Airbyte to connect to an AWS RDS Postgres database as its configuration database. Any thoughts on where to go to try and sort out what we don't have configured correctly?

2021-08-12 18:04:49 INFO i.a.s.ServerApp(getServer):171 - {workspace_app_root=/workspace/server/logs} - Creating config repository...
2021-08-12 18:04:50 INFO i.a.c.p.ConfigPersistenceBuilder(getDbPersistenceWithFileSeed):93 - {workspace_app_root=/workspace/server/logs} - Creating db-based config persistence, and loading seed and existing data from files
2021-08-12 18:04:50 INFO i.a.c.p.FileSystemConfigPersistence(createWithValidation):62 - {workspace_app_root=/workspace/server/logs} - Constructing file system config persistence (root: /configs)
2021-08-12 18:04:50 INFO i.a.c.p.FileSystemConfigPersistence(createWithValidation):71 - {workspace_app_root=/workspace/server/logs} - Config volume is ready (waiting time: 0 s)
2021-08-12 18:04:50 INFO i.a.c.p.ConfigPersistenceBuilder(getDbPersistence):103 - {workspace_app_root=/workspace/server/logs} - Use database config persistence.
2021-08-12 18:04:50 INFO i.a.c.EnvConfigs(getEnvOrDefault):302 - {workspace_app_root=/workspace/server/logs} - CONFIG_DATABASE_USER not found or empty, defaulting to xxxxxxxx
2021-08-12 18:04:50 INFO i.a.c.EnvConfigs(getEnvOrDefault):302 - {workspace_app_root=/workspace/server/logs} - CONFIG_DATABASE_PASSWORD not found or empty, defaulting to xxxxxxxx
2021-08-12 18:04:50 INFO i.a.c.EnvConfigs(getEnvOrDefault):302 - {workspace_app_root=/workspace/server/logs} - CONFIG_DATABASE_URL not found or empty, defaulting to jdbc:<postgresql://xxxxx.xxxxxx.us-east-1.rds.amazonaws.com:5432/airbyte>
2021-08-12 18:04:50 WARN i.a.d.Databases(createPostgresDatabaseWithRetry):55 - {workspace_app_root=/workspace/server/logs} - Waiting for database to become available...
2021-08-12 18:04:50 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$3):128 - {workspace_app_root=/workspace/server/logs} - Testing airbyte configs database connection...
2021-08-12 18:04:54 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):60 - {workspace_app_root=/workspace/server/logs} - Database is not ready yet. Please wait a moment, it might still be initializing...

Hi, I am a happy user of Airbyte and recently got buy-in from management to use it as our main elt-tool. However, our security guys are not super happy about the Kubernetes version is in Beta - no special functionality is missing - it frankly seems to be the label/idea of being in Beta. So, my kind of dull question follows: is there an estimation/roadmap on when Airbyte Kubernetes will have a stable version? Thanks for an awesome product!