just bumping my last question...what are the actual user perms needed for a database user in redshift? They're not documented but I've found a bunch of posts in here about needing

create database

and I was hoping to get an inventory of the perms actually needed

Hey folks, I do have question about kafka connector available in Airbyte. Our kafka cluster support SSL, therefore I don't see an option on the connector properties to select the keystore and truststore, any idea how I can get the connector to support SSL? do I have to modify the current connector implementation? thanks! looking forward to get an answer soon.

Does anyone have experience deploying airbyte opensource in AWS Fargate? I have a developer friend/collegue struggling to accomplish. In the long run, we are looking to set up something that scales based on jobs.

Hi Team i am trying to setup airbyte on Minikube to test it out, while doing an

kubectl apply -k kube/overlays/stable

I get this error

The Job "airbyte-bootloader" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"a9791ea0-5c6b-4896-ab01-23fff164bca1", "job-name":"airbyte-bootloader"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"airbyte-bootloader-container", Image:"airbyte/bootloader:0.40.17", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"AIRBYTE_VERSION", Value:"", ValueFrom:(*core.EnvVarSource)(0x401169b020)}, core.EnvVar{Name:"DATABASE_HOST", Value:"", ValueFrom:(*core.EnvVarSource)(0x401169b040)}, core.EnvVar{Name:"DATABASE_PORT", Value:"", ValueFrom:(*core.EnvVarSource)(0x401169b060)}, core.EnvVar{Name:"DATABASE_PASSWORD", Value:"", ValueFrom:(*core.EnvVarSource)(0x401169b080)}, core.EnvVar{Name:"DATABASE_URL", Value:"", ValueFrom:(*core.EnvVarSource)(0x401169b0a0)}, core.EnvVar{Name:"DATABASE_USER", Value:"", ValueFrom:(*core.EnvVarSource)(0x401169b0c0)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0x40073879f8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0x400fc68480), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil)}}: field is immutable

any inputs i have the latest changes on my local

Hi Folks! I am trying to setup airbyte-temporal and I have already setup one airbyte metadb as an external pg db. For temporal, the pod is specifically looking for temporal & temporal_visibility DBs. I have 2 more new DBs created but their name is slightly different, along with diff creds. Is there a way to override this? I was checking this script which runs by default. https://github.com/airbytehq/airbyte/blob/master/airbyte-temporal/scripts/update-and-start-temporal.sh I was thinking that I’ll have to modify this to suit my use case, can someone please guide me to the right path? Thanks :)

Anyone using the Google Search Console connector have issues with the streams being empty? It seems like I am only getting data from 3 streams

search_analytics_by_date
sites
sitemaps

and the rest of the streams are empty from the sync

search_analytics_all_fields
search_analytics_country
search_analytics_page
search_analytics_device
search_analytics_query

There are no failures indicated in the log

failures: [ ]

Folks, I am trying to create MS SQL Server to S3 and try to move the data. I see nothing wrong with the logs, but still the data did not come in. Any idea why? 2022-10-25 190739 source > Internal schemas to exclude: [spt_fallback_db, spt_monitor, cdc, spt_values, INFORMATION_SCHEMA, spt_fallback_usg, MSreplication_options, sys, spt_fallback_dev] 2022-10-25 190740 source > Table test_jaydebeapi column ID (type int[10]) -> false 2022-10-25 190740 source > Table test_jaydebeapi column NAME (type varchar[1]) -> false 2022-10-25 190740 source > using CDC: false 2022-10-25 190740 source > Closing database connection pool.

Hi All. I have a scenario where I need to connect to same endpoint using different IDs as request param. Response for all of these IDs is same and need to be persisted in same table at our end. To solve this problem we have come up with two approaches: 1. Create a simple connector that would fetch the response for each ID passed as parameter and write to target table in Append mode. Loop through the IDs at orchestration layer. This approach is working. 2. Pass the list of IDs as parameter to the airbyte and based on number of IDs the looping should happen inside the Airbyte connector. For this I have defined one class and JSON schema is passed as via config file. This only works for first element and the job goes to success. For my testing I passed two IDs and also defined two class one for each ID. I am able to get the response but on UI this would mean writing to two separate output.(following the example of Customer and Employee stream) Is there a way I can have only one stream defined in the connector code and can loop through the same stream(using different params) inside connector code?

Hi everyone!! has anyone ever encountered the following error creating a connection to Snowflake, from an installation on Kubernetes, from a pod doing a telnet to the account and port 443 works fine. The example String conn account.us-east-1.snowflakecomputing.com

Hello Everyone, This is my first time using Airbyte. I have question about facebook Gcp integration. I am fetching only Ads, Adsets, AdInsights Stream from Facebook Marketing. with the start date as 1 Jan 2022. After the initial sync up when I checked the records in GCP. I see all the ads and adset info starting from Jan 1 2019. (The day I start using facecbook marketing). and Ads Insights from 1 Jan 2022. Can anybody please clear this out. Why the start date does not apply on Ads, and Adset stream?

Hi Everyone, Wanted to know if anyone is building or there are plans to build a unified schema for various connectors using airbyte. Something similar to example https://www.rutter.com

Hi everyone. I'm trying to use Kafka topic as a source. Test passed with one warning

WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword

Topic contains fresh messages but sync returned "no records"

2022-10-26 06:24:14 WARN i.a.w.g.DefaultReplicationWorker(run):305 - State capture: No new state, falling back on input state: io.airbyte.config.State@2820a8af[state={}]
2022-10-26 06:24:14 INFO i.a.w.g.DefaultReplicationWorker(run):316 - sync summary: {
  "status" : "completed",
  "recordsSynced" : 0,
  "bytesSynced" : 0,
  "startTime" : 1666765438208,
  "endTime" : 1666765454843,
  "totalStats" : {
    "recordsEmitted" : 0,
    "bytesEmitted" : 0,
    "sourceStateMessagesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "recordsCommitted" : 0,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "replicationStartTime" : 1666765438208,
    "replicationEndTime" : 1666765454843,
    "sourceReadStartTime" : 1666765438243,
    "sourceReadEndTime" : 1666765452324,
    "destinationWriteStartTime" : 1666765438295,
    "destinationWriteEndTime" : 1666765454842
  },
  "streamStats" : [ ]
}

What should I do to detect the problem?

Hey folks, I do have question about kafka connector available in Airbyte. Our kafka cluster support SSL, therefore I don't see an option on the connector properties to select the keystore and truststore, any idea how I can get the connector to support SSL? do I have to modify the current connector implementation? thanks! looking forward to get an answer soon.

Hello everyone!!! can you help me please, I have the following error creating a connection to Snowflake from a Kubernetes installation, from a pod doing a telnet to the account and port 443 from snowflake works fine, but from Airbyte it doesn’t, I have version 0.40.17, using the following chart https://github.com/airbytehq/airbyte/blob/v0.40.33-helm/charts/airbyte/Chart.yaml. The example string conn <account>.us-east-1.snowflakecomputing.com. Thank you for your help

hey folks 👋

quick question: I'm using the clickhouse destination to move some data and i've found airbyte is moving all data into a

public

database inside clickhouse instead of the database I indicated in the destination config page. Is this because the connector is still in alpha?

SQLState: S0002, Message: Could not allocate space for object 'dbo.SORT temporary run storage: 140739843915776' in database 'tempdb' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup. help my source is mssql

Is there a way to configure the number of rows that are pulled from the source at one time? Our instance is only pulling 1000 rows at a time and the replication step is taking up much more time than we would hope.

I'm attempting an initial sync on Airbyte OS, using the Google Analytics (Universal Analytics) connector version 0.1.30. The sync ran for about 12 hours and then failed with the error shown in the code snippet at bottom: My authorization method = service account key, which in prior (shorter) attempts has never failed for me with

ACCESS_TOKEN_EXPIRED

. The 2nd sync attempt started right after the failure and is still going, so it's not clear that the token actually expired, or really that the token ever expires. The service account JSON key is formatted like below, with sensitive values redacted. I think my recourse is to chunk up the initial sync into pieces which take less time, then stitch together the results and reload the target table from there, and hope this doesn't confuse incremental sync. Thoughts?

{
  "type": "service_account",
  "project_id": "XXXXX",
  "private_key_id": "XXXXX",
  "private_key": "-----BEGIN PRIVATE KEY-----XXXXX-----END PRIVATE KEY-----\n",
  "client_email": "XXXXX",
  "client_id": "bigint",
  "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
  "token_uri": "<https://oauth2.googleapis.com/token>",
  "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
  "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/XXXXX>"
}

Hi, guys! I've been trying to make a connection between airbytes and google ads for the first time and I am really stuck in this problem: I can't seem to find these tokens -> client id, client secret, refresh token and customer id! the only thing I have access to is my developer token! So if anyone knows how to acess the rest of the tokens I'd be very grateful! here's a screenshot of the airbytes interface:

Hello, I just tried syncing another database from Mongo Atlas but got back error:

Invalid $project :: caused by :: '$' by itself is not a valid FieldPath

so i am running on a forked repo of the airbyte repo. If i want to upgrade, what else is required other then changing the image version tags in the

kustomization.yaml

file? do i need to port over

airbyte/kube/resources

to my own resource folder (

deployments/prd/us_east_2/resources

)? I have prod and staging deployment folders setup and deploying via flux.

I've been trying to connect MS SQL on Airbyte running on EC2 instance. The connection failed when the host name is used but succeeded with the IP address. Does anyone know why host name can't build the connection?

Hi All , I am trying to install Airbyte first time, I am new to Airbyte. Here is the error I am getting when I copy from git using

git clone <https://github.com/airbytehq/airbyte.git>

. I would appreciate any help

Hi there! Could someone help me understand whether using a single CDC PostgreSQL source with multiple destinations is a valid usecase? What I don’t quite get is what’s going to happen to the pg replication slot after it will be emptied into the first destination? Will Airbyte somehow fan out the changes into all destinations at once, or will it simply not see any delta for every subsequent destination?

Folks, I'm unable to get a Salesforce object

Lead

due to something w

<https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/source.py>

. with the text search i'm getting

leadstatus

,

leadshare

but not

Lead

specifically. Its only happening with this particular object. When I login to sfdc with the same credentials I do see this object and able to query fine, but its the airbyte connector runs where its not working, it doesnt find this object at all. Any leads with what I could be trying next to debug?

Is there a way to perform airbyte dbt-transformations server-less. If yes then how should I go about it and what tools can I use more to save my compute resources.

Hello everyone, is there something I could configure in terms of detecting those stuck syncs?

Hi everyone, just wondering how you would approach this: We have sources that take a long time to sync. To save us from running Redshift basically 24/7, we’re going to dump the streams into Parquet files in S3 first as a data lake. My question is: how would you get those files into Redshift? I looked at the S3 connector, but this would require setting up a source and connection per synced stream into Redshift, which seems wildly inefficient. I could do a Lambda function, but thought I’d see if there are any other options floating around before I start coding something custom.

Hello everyone, is there any solution to reset the configuration data of airbyte but leave the connections, source and destination configuration that I've already added?