Hey all, I couldn’t figure out if Airbyte supported this use case, so wanted to check here. I want to sync multiple sources of the same type to the same destination table, along with some custom columns for which source it came from. For example, I’d want to connect to 10 different salesforces, and have all of them go to the same postgres tables (instead of having 10 sets of tables) — but this of course means I’d need additional columns for each table specifying which salesforce instance it came from. Is this possible?

Hello everyone, I'm facing one issue in transferring data from SQL Server to BigQuery using AirByte. datetime data is not transferred correctly in bigquery. In SQL Server, datatype is datetime2. Can anyone help to resolve this issue? Original datetime value is '2020-05-06 174444.6185145' and transferred in bigquery is '1588787084618514500'

Hi all, was wondering if Airbyte support Postgres RDS Warehouse as a destination

Hello all, is anyone able to tell me the difference in instance types with relation to deploying on AWS? What's the difference between t2.micro vs t2.medium from Airbyte's POV?

Hello. I created a connection between Google Analytics and BigQuery. When I select raw data, the sync works and I can see the json data in the BigQuery tables. However, when I select "Basic normalization" in the settings, my tables are empty. Could you please advise? Thanks!

Hello, I'm implementing a custom source based on the HTTP Python generator and I need to know which parameters has been used during the previous run (to know what start_date/end_date to use). I feel like I should use stream states but I can't find a good example of how it works in the codebase, can somebody help me? Thanks!

Hi All, i just created a new workspace by api call on my local Airbyte server. However, i can’t navigate to the new workspace on the UI. How do I see the new workspace i made?

Hi all, I hope someone can help point me in the right direction. Is there a way to make namespaces for connections, sources and destinations in Airbyte, so that I can isolate them from each other? Background: we have multiple teams that work with the same types of sources (eg. 10 teams running facebook ads from different accounts) and destinations (10 distinct data warehouse instances). I would like to isolate the sources and destinations belonging to team 1 from those belonging to team 2, so that I don’t accidentally make a new connection between the wrong source and destinations

Hi, I'm trying to figure out a "not quite prod, but better than dev" deploy structure. I have a couple of questions: 1. It looks like the scheduler and server use the most memory. Does it make sense to put those on beefier machines? I assume that they can not be duplicated so one machine for each (or both on one). 2. Is there a way to get the workers to start on a different machine? In other words, is there a worker pool I can start or is there a way to use bin-packing to get the most out of the worker machine?

hi everyone! I'm exploring Airbyte as a possible ELT solution for my company. I was able to successfully set up Airbyte locally and create my desired connections locally. Now, I'm moving to a more complex structure and I have a question: is it possible to configure Airbyte using an infrastructure-as-code strategy? I know that I can easily create the connections in the UI, but the entire infrastructure of my company is versioned using Terraform, so we can easily create new Kubernetes clusters. How can I do that with Airbyte? 😄

Hi all, I’m testing airbyte deployment with GKE autopilot and it seems like HostPort/pod affinity is not permited in autopilot as the error below says.

kubectl apply -k kube/overlays/stable 

serviceaccount/airbyte-admin created
<http://role.rbac.authorization.k8s.io/airbyte-admin-role|role.rbac.authorization.k8s.io/airbyte-admin-role> created
<http://rolebinding.rbac.authorization.k8s.io/airbyte-admin-binding|rolebinding.rbac.authorization.k8s.io/airbyte-admin-binding> created
configmap/airbyte-env-m2467g889g created
configmap/airbyte-temporal-dynamicconfig created
configmap/sweep-pod-script created
secret/gcs-log-creds created
service/airbyte-db-svc created
service/airbyte-minio-svc created
service/airbyte-server-svc created
service/airbyte-temporal-svc created
service/airbyte-webapp-svc created
deployment.apps/airbyte-db created
deployment.apps/airbyte-pod-sweeper created
deployment.apps/airbyte-scheduler created
deployment.apps/airbyte-temporal created
deployment.apps/airbyte-webapp created
persistentvolumeclaim/airbyte-minio-pv-claim created
persistentvolumeclaim/airbyte-volume-configs created
persistentvolumeclaim/airbyte-volume-db created
persistentvolumeclaim/airbyte-volume-workspace created
Error from server ([denied by autogke-no-host-port] container airbyte-minio specifies a host port; disallowed in Autopilot.  error when creating "kube/overlays/stable": admission webhook "validation.gatekeeper.sh" denied the request: [denied by autogke-no-host-port] container airbyte-minio specifies a host port; disallowed in Autopilot. 
Error from server ([denied by autogke-pod-affinity-limitation] pod affinity is not allowed in Autopilot on topologyKeys: <{"kubernetes.io/hostname"}>; Autopilot allows pod affinity only on topologyKeys: <["topology.kubernetes.io/region", "topology.kubernetes.io/zone", "failure-domain.beta.kubernetes.io/region", "failure-domain.beta.kubernetes.io/zone"]>.  error when creating "kube/overlays/stable": admission webhook "validation.gatekeeper.sh" denied the request: [denied by autogke-pod-affinity-limitation] pod affinity is not allowed in Autopilot on topologyKeys: <{"kubernetes.io/hostname"}>; Autopilot allows pod affinity only on topologyKeys: <["topology.kubernetes.io/region", "topology.kubernetes.io/zone", "failure-domain.beta.kubernetes.io/region", "failure-domain.beta.kubernetes.io/zone"]>.

Are there any workarounds on this issue?

Hey, we are currently gathering experience with our first connection (custom connector > redshift). We can’t find much information on how the namespace customization works. Is it possible to inject own variables into the custom format as ${SOURCE_NAMESPACE}? Are there other variables that are offered by you in the default?

Hello, I have set up airbyte for the first time with a Postgres > BigQuery connector, and I’m facing a few issues and I’m not sure whether I’m doing it right, or if they are expected • I select only one table in the source schema, but once I saved I can’t see anymore the “not selected” ones, and when I update the source schema all tables get selected so I lose the information of which tables were previously selected • I get the following log at the end of the sync (though successful) :

2021-07-23 17:08:06 INFO () EnvConfigs(getEnvOrDefault):302 - WORKER_ENVIRONMENT not found or empty, defaulting to DOCKER
2021-07-23 17:08:06 INFO () DefaultNormalizationWorker(run):77 - Normalization executed in 0.
2021-07-23 17:08:06 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling...
2021-07-23 17:08:06 ERROR () Exceptions(swallow):84 - Swallowed error.
java.lang.NullPointerException: null value in entry: sync_cpu_request=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:32) ~[guava-30.1.1-jre.jar:?]
	at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:171) ~[guava-30.1.1-jre.jar:?]
	at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:281) ~[guava-30.1.1-jre.jar:?]
	at io.airbyte.scheduler.persistence.job_tracker.TrackingMetadata.generateSyncMetadata(TrackingMetadata.java:62) ~[io.airbyte.airbyte-scheduler-persistence-0.27.5-alpha.jar:?]
	at io.airbyte.scheduler.persistence.job_tracker.JobTracker.generateSyncMetadata(JobTracker.java:211) ~[io.airbyte.airbyte-scheduler-persistence-0.27.5-alpha.jar:?]
	at io.airbyte.scheduler.persistence.job_tracker.JobTracker.lambda$trackSync$3(JobTracker.java:134) ~[io.airbyte.airbyte-scheduler-persistence-0.27.5-alpha.jar:?]

Thanks !

Hello team!, we at DataKubes have created the first Data Management Platform, we are integrating Airbyte to the DataKubes orchestrator, we where refered to you by Zohan from mindsb, is there a docker image release for Airbyte?

I am using airbyte for mysql binlog incremental data and facing issue (Exception in thread "main" tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro:) could anyone help me on this.

hi everyone! I have two questions: 1. Is it possible to use a local DBT project as a transformation? If so, how can I do that? Which path should I provide? 2. When we have multiple data sources, what is the best practice for transformations? Each data source should have it's own DBT project? Or a single DBT project that is triggered for every data source?

Question about Postgres normalization. I’ve specified some fields in the stream spec as

date-time

but they appear in the destination as

varchar

. Do i need to add my own dbt transformation to fix?

hey everyone 🙂 I'm trying to setup my custom DBT transformations for my connection but not being able to do so. I'm runnning Airbyte on Kubernetes. This is my transformations project: https://github.com/delucca-workspaces/analytics/tree/feat/user-access/dbt This is the repository URL I'm using inside Airbyte: https://github.com/delucca-workspaces/analytics.git This is the branch name I'm using in my custom DBT transformer inside Airbyte:

feat/user-access

This is the command I'm using for DBT inside Airbyte:

run --project-dir dbt

(if anyone needs it, this is the link directly to the branch: https://github.com/delucca-workspaces/analytics/tree/feat/user-access) I've attached the logs. In a nutshell, it fails after a few minutes. I've tried running

dbt run --project-dir dbt

locally (inside the root of my repository) and it works. Only in Airbyte it fails

I have a connector with 10 tables and one of them i want to perform a full refresh. 1. what is the easiest way to achieve this? 2. If i delete all related tables in the database e.g.

mytable

,

mytable_scd

,

_airbyte_raw_mytable

. and resync the connector, will it perform a full-refresh?

Hi, I had a quick question I wanted to ask, I deployed airbyte on a compute engine instance and connected the cloud proxy on it, however even though I connected with my db replica, the CPU utilization for my database has immensely increased (till 80%). I wanted to understand how airbyte is connecting with my replica SQL instance since this is unexpected behavior. Secondly, to make use for CDC it was written in the docs to upgrade postgres to 10+, would CDC also work on postgres 9.6?

Is there a way to change how the tables are created in Snowflake? I have a role that is only allowed access to a few tables in my schema and I’m using Full Refresh | Overwrite. I suspect it’s using

CREATE OR REPLACE

because every time the data syncs I have to re-grant the

SELECT ON TABLE

. Is there a way to change this or update the create to add

COPY GRANTS

Hi everyone, i just want to ask a question related to data pulling in airbyte: I have encounter one problem related to data pulling like when we pull huge data from any source to any destination we never seen a minor mistake related to row missing but i have check with few rows of data and have seen that always i get one row less from the total ,suppose we have 5 rows in source and in destination we always get n-1 rows. can any one encounter this situation till now? or any one tell be what will be the reason?

Hi 👋 last week I asked about workspaces and when/how that would become available. I was happy to learn from the answer that a pull request with the required changes to switch to a workspace other than default was already in the making. In fact, this pull request has been merged to master this week (link to thread for more background: https://airbytehq.slack.com/archives/C021JANJ6TY/p1626896215315900) In the latest Airbyte version from Github, I can create a new workspace through the API, but I haven’t yet found a way to switch to that workspace. How can I switch between workspaces through the OSS UI or api ?

Hey Guys, has anyone been able to make the postgres CDC work. Mine seems to be getting stuck at "testing connection" . But not getting any error

Also is there any plan for CDC or SQL and Oracle

Hi Everyone, Anyone can tell how much data volume airbyte handel easily ?

Anyone here use Heap analytics? Would love to see a connector for that 🙂

Hi Team, where can i find the list of sources which supports Incremental - source_defined_cursor?

Hi team, is there a way to reset a specific table in a connector (without resetting all the tables in the ocnnector)?

Can you run airbyte on multiple instances behind a load balancer? The database will run on AWS RDS Postgres. Is this architecture possible?