Will there any issues on Snowflake Destination on Airbyte due to

OCSP and TLS impacts from Snowflake load balancer (Envoy) migration

•

"Start datetime" is set to "user input" to allow the user of the connector configuring a Source to specify the time to start syncing

what it means in airbyte UI

@kapa.ai is abctl recommended for installation on Prod environment ?

@kapa.ai One of the sync was running for more than 12 hours with following logs repeating:

2024-09-02 08:31:35 destination > 2024-09-02T08:31:35,556`pool-3-thread-1`17`INFO`i.a.c.i.d.b.BufferManager(printQueueInfo):118 - [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-09-02 08:31:35 destination > 2024-09-02T08:31:35,627`pool-5-thread-1`19`INFO`i.a.c.i.d.FlushWorkers(printWorkerInfo):143 - [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0

What is the cause of this?

@kapa.ai nginx: [emerg] host not found in upstream “airbyte-api-server” in /etc/nginx/nginx.conf:96

Hi everyone, I am using airbyte to fetch all the ad performance from meta. I want to parse all the creatives properly. Each type of creative is being stored in a different format in meta tables: • Single image ad • Carousel (image / video) • Flexible ads • Dynamic creative ads • Catalogue ads • Placement customisations • Partnership ads • Manual upload vs using existing post It’s getting difficult for me to figure out a robust parsing logic. Can anyone help me out? If you have used motion / triple whale for creative analytics, we want a similar outcome.

@kapa.ai Any table i am trying to refresh in Airbyte is going in Queued state and not syncing how can i diagnose the issue ?

@kapa.ai I have multiple pipelines setup on my open source airbyte , which pulls the data from a s3 bucket and posts it to db , now i want to make a staging instance for the whole airbyte .. guide me throught the processs

how to add elasticsearch source into airbyte api

Why for Loading Method for BQ destination GCS Staging is recommended instead of standard batched inserts?

Uploading Docker-based custom connectors with abctl local install and self hosted registry

I am using airbyte-ci to build a custom connector on Mac. This connector will be used by Airbyte on GKE. When pulling the image from the Airbyte UI, the pod has this error :

exec /usr/bin/sh: exec format error

Any idea how to fix this ?

Write me terraform to create all resources to have setup big query destination with GCS staging

In airbyte, can I make basic transformations/normalization without using dbt?

i have a self hosted Airbyte with a Google Ads to Snowflake connection. The synchronisation takes several hours, due to Google Ads API limits. During this time, there are continually sql queries ran on snowflake, hence quite a high run cost for snowflake. Is there a way to make Airbyte extract data from Google Ads, store it in a S3 bucket and then upload it to snowflake once it's all extracted?

After updating to the latest version we get this exception when starting the worker:

2024-09-03 11:10:19 ERROR i.m.r.Micronaut(handleStartupException):349 - Error starting Micronaut server: Failed to inject value for parameter [processFactory] of class: io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl

Message: No bean of type [io.airbyte.workers.process.ProcessFactory] exists. 
* [ProcessFactory] is disabled because:
 - None of the required environments [k8s] are active: [control-plane]
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> ActivityBeanFactory.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> new CheckConnectionActivityImpl(WorkerConfigsProvider workerConfigsProvider,[ProcessFactory processFactory],SecretsRepositoryReader secretsRepositoryReader,Path workspaceRoot,AirbyteApiClient airbyteApiClient,String airbyteVersion,AirbyteMessageSerDeProvider serDeProvider,AirbyteProtocolVersionedMigratorFactory migratorFactory,FeatureFlagClient featureFlagClient,GsonPksExtractor gsonPksExtractor,WorkloadClient workloadClient,WorkloadIdGenerator workloadIdGenerator,MetricClient metricClient,ActivityOptions activityOptions,LogClientManager logClientManager)
io.micronaut.context.exceptions.DependencyInjectionException: Failed to inject value for parameter [processFactory] of class: io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl

Message: No bean of type [io.airbyte.workers.process.ProcessFactory] exists. 
* [ProcessFactory] is disabled because:
 - None of the required environments [k8s] are active: [control-plane]
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> ActivityBeanFactory.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> new CheckConnectionActivityImpl(WorkerConfigsProvider workerConfigsProvider,[ProcessFactory processFactory],SecretsRepositoryReader secretsRepositoryReader,Path workspaceRoot,AirbyteApiClient airbyteApiClient,String airbyteVersion,AirbyteMessageSerDeProvider serDeProvider,AirbyteProtocolVersionedMigratorFactory migratorFactory,FeatureFlagClient featureFlagClient,GsonPksExtractor gsonPksExtractor,WorkloadClient workloadClient,WorkloadIdGenerator workloadIdGenerator,MetricClient metricClient,ActivityOptions activityOptions,LogClientManager logClientManager)

hey @kapa.ai do you know how to use datadog to monitor airbyte deployed using abctl

How to use multiple fields from the parent stream in a SubstrreamPartitionRouter setup in the childstream within the manifest.yml?

@kapa.ai I want to use airbyte builder to ingest data from an api. The api endpoint accepts only start and end date and provide aggregated data. There is no possibility of grouping by date. However, I want to ingest past data grouper by month for example. Is it possible to do such iteration in builder? For example: I want to have data from 2024-01-01, until this month but in monthly interval.

@kapa.ai How do I restart the Airbyte process once deployed to Kubernetes now that docker compose is deprecated

What is metrics "readFromDest" (I was only expecting a "readFromSource" and "writeToDest") ?

How to change

MAX_SYNC_WORKERS

on running airbyte instance (installed by abctl)

@kapa.ai is dbt transformation tab available on OSS?

I have a JSON results with this exampl:

[{
    "USD": 1,
    "AED": 3.6725,
    "AFN": 70.5031,
    "ALL": 90.2195,
    "AMD": 387.6506,
    "ANG": 1.79,
}]

is it possible to modify the schema to only have 2 columns? 1st column will be currency and the 2nd column will be value

@kapa.ai Is there anyway to get the error messages for a failed job via the Airbyte API?

@kapa.ai how do I migrate a docker compose production deployment to abctl?

@kapa.ai what happened to the CloudFlare R2 destination? It is no longer in Airbyte Cloud.

what does this mean? Executing soft-reset on final table of stream StreamConfig

Are there recommendations for "airbyte-server" memory limits? my instances (2 pods) are growing up to 10gb each

what does this mean “unable to install airbyte chart: unable to install helm: pre-upgrade hooks failed: 1 error occurred”