Hello, I'm using the Helm chart to deploy airbyte. Is there a way to customize the image for

airbyte/normalization:0.4.0

? I'm using an ARM processor and I'm thinking I have to recompile something for it. The error message I get in its log is

exec /bin/sh: exec format error

Trying to setup AWS Secrets Manager, and added the values in the .env file for

SECRET_PERSISTENCE

,

AWS_ACCESS_KEY

, and

AWS_SECRET_ACCESS_KEY

based on this documentation: https://docs.airbyte.com/operator-guides/configuring-airbyte/ but I’m getting this error when I bring Airbyte back up

| 2023-05-22 13:56:11 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type  [io.airbyte.config.persistence.split_secrets.SecretsHydrator]
airbyte-worker                    |
airbyte-worker                    | Message: awsAccessKey cannot be empty, to use a default region call AWSSecretManagerPersistence.AWSSecretManagerPersistence()
airbyte-worker                    | Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> List.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> new CheckConnectionActivityImpl(WorkerConfigsProvider workerConfigsProvider,ProcessFactory processFactory,[SecretsHydrator secretsHydrator],Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,AirbyteApiClient airbyteApiClient,String airbyteVersion,AirbyteMessageSerDeProvider serDeProvider,AirbyteProtocolVersionedMigratorFactory migratorFactory,FeatureFlags featureFlags) --> SecretsHydrator.secretsHydrator([SecretPersistence secretPersistence])
airbyte-worker                    | io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.config.persistence.split_secrets.SecretsHydrator]
airbyte-worker                    |
airbyte-worker                    | Message: awsAccessKey cannot be empty, to use a default region call AWSSecretManagerPersistence.AWSSecretManagerPersistence()

Hey! I'm trying to set up Airbyte

0.45.17

on GKE (using helm). Not sure why syncing fails after around

27m 30s

with errors like

Failure during reporting of activity result to the server. ActivityId = .... , ActivityType = RunWithJobOutput, WorkflowId=connection_manager_7c51cd55-1d52-4402-a457-4ea954a2274d, WorkflowType=ConnectionManagerWorkflow, RunId=... io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed

Any way to increase the timeout? I also can see many pods being in

Init:Error

or

Evicted

state... So it looks like syncs are competing for CPU requests. Any way to limit the number of parallel jobs?

Hi everyone 👋 Has anyone deploy Airbyte on AWS Fargate Spot? I would like to know how to deal with stop instance and how to configure

🔥 Community Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-bootloader At 1pm PDT click here to join us on Zoom!

Hello! Im trying to use the nodeSelector for my jobs (im deploying using Helm in kubernetes on GCP) but it simply doesnt work

Hi Guys, I'm relatively new at Airbyte, have it running in EC2, pulling data from various Oracle/Postgres/Custom sources into Snowflake. Up until now this has been a POC and I haven't needed to think too much about the infrastructure side, but now we are looking to push it into production I'd like some advice 🙂 Background: I will be deploying: • On AWS EC2/Docker • 50+ Source connectors (35 Oracle, 15 Custom via API/JSON, 3 Postgres • 1 Destination connector (Snowflake) • Each of these connections will be staggered (mostly to not overwhelm the source vCPU) 1. Is there a preferred instance family best suited for Airbyte (currently I have it on the T family)? 2. Re resourcing, I'm unclear how to calculate the Disk & CPU required based on existing pipelines. I have some existing connections which have ran I can calculate from if I know a basic formula - is there such a thing? 3. So far I have used the in-docker Postgres database. Is there any way to port across existing config/state from the local machine (other than some clunky export/import)?

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-bootloader At 16:00 CEST / 10am EDT click here to join us on Zoom!

Can lead me to a description how write custom report in source-data-api?

does anyone have a

.env

file they can share that shows how to configure? i am getting issues with micronaut and im pretty sure its because i only have these two, but the docs are not the best:

WORKSPACE_ROOT=/tmp/workspace/
WORKSPACE_DOCKER_MOUNT=airbyte_workspace

Hi, I've deployed Airbyte on EKS. I know Airbyte supports OpenTelemetry to collect metrics that can forward metrics to Prometheus. I wanted to know if Prometheus scraping can be directly supported (without deploying OpenTelemetry)?

Is the airbyte server a STS in the helm chart? I currently had issues with the airbyte server restarting, making it unable to mount the airbyte volume configs, so I have reversed engineered the chart, deploying vanilla, and changed it to a STS, version

0.44.4

. If anyone seen this issue?

how do we implement user authentication (google) when entering the application?

When I deploy Airbyte on EKS, the deployment creates a Kubernetes Secret

<deployment>-airbyte-secrets

. This secret does not have a

DATABASE_PASSWORD

key. How can I modify this secret to also contain password secret?

keep getting this one

2023-05-24 17:25:18 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type  [io.airbyte.db.check.DatabaseMigrationCheck]

Path Taken: new ApplicationInitializer() --> ApplicationInitializer.configsDatabaseMigrationCheck --> DatabaseMigrationCheck.configsDatabaseMigrationCheck(DSLContext dslContext,[Flyway configsFlyway],String configsDatabaseMinimumFlywayMigrationVersion,Long configsDatabaseInitializationTimeoutMs)
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.db.check.DatabaseMigrationCheck]

Does anyone else's airbyte instance intermittently crash during a sync? Airbyte version: 0.44.0 Container: docker Platform: Google Compute, n2-standard-4 (16GB memory)

how much ram and cpu is ideal for airbyte for syncing 80 gb of data ?

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals At 1PM PDT click here to join us on Zoom!

what can be the reason of a sync to suddenly fail without any clear reason ?(k8s, 0.40.32)

2023-05-24 19:35:38 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-05-24 19:35:38 WARN i.a.w.s.LauncherWorker(killRunningPodsForConnection):253 - There are currently running pods for the connection: [orchestrator-repl-job-25900-attempt-0]. Killing these pods to enforce one execution at a time.
2023-05-24 19:35:38 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):256 - Attempting to delete pods: [orchestrator-repl-job-25900-attempt-0]
2023-05-24 19:35:38 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):261 - Waiting for deletion...
2023-05-24 19:35:39 WARN i.a.w.s.LauncherWorker(killRunningPodsForConnection):253 - There are currently running pods for the connection: [orchestrator-repl-job-25900-attempt-0]. Killing these pods to enforce one execution at a time.
2023-05-24 19:35:39 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):256 - Attempting to delete pods: [orchestrator-repl-job-25900-attempt-0]
2023-05-24 19:35:39 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):261 - Waiting for deletion...
2023-05-24 19:35:39 INFO i.a.w.p.AsyncOrchestratorPodProcess(computeExitValue):129 - State Store missing status. Orchestrator pod orchestrator-repl-job-25900-attempt-0 non-existent. Assume failure.
2023-05-24 19:35:39 INFO i.a.w.s.LauncherWorker(lambda$run$3):227 - Destroying process due to cancellation.
2023-05-24 19:35:39 WARN i.a.w.p.AsyncOrchestratorPodProcess(destroy):186 - Wasn't able to delete pod orchestrator-repl-job-25900-attempt-0 from namespace default
2023-05-24 19:35:39 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):316 - Stopping temporal heartbeating...

Is there any way to configure a postgres source to only pick data after a certain update value?

what are spec worker, connection check worker, sync worker, discovery worker ?

How do I set up basic authentication in my Airbyte deployment in my Kubernetes cluster - (I am using Helm to deploy)? I tried setting BASIC_AUTH_USERNAME, and BASIC_AUTH_PASSWORD as env variables in my webapp and server container but still, there is no use. How do I secure access to my Airbyte application in my Kubernetes cluster?

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

Hey guys! I've been messing with Airbyte for some time and I have the following scenario: I deployed the tool on GKE and the pv of the database is filling up very fast (1GiB/Day), I believe this is because of log storage (I left it default, I didn't put S3 or GCS). When I had this problem in the VM, I configured logrotate to delete logs older than 7 days and it solved the problem. But I'm not finding a similar parameter in GKE, has anyone had to deal with this? In VM I used this parameter (TEMPORAL_HISTORY_RETENTION_IN_DAYS)

Hi there, in a OSS install of airbyte how to switch between workspaces?. If we have created multiple workspaces using api

What should I set

SPEC_CACHE_BUCKET

to? It says it defines the bucket for caching specs and speeds up spec operations, but I don't know what that means 😄 https://docs.airbyte.com/operator-guides/configuring-airbyte/

Hello everyone! ❔ airbyte I want to run Airbyte on multiple different EC2 servers (so using the EC2 pulling the docker-compose file approach like here), but using some selfwritten connectors. My main headache right now is how to manage the code for those connectors. Anyone done something similar and could share their experiences on the approach? • Do you Git fork the Airbyte repository and add there whatever connectors and settings are needed and pull/deploy that to the specific servers • Do you have for each connector an own repository and push the image to Dockerhub for example and just use it in Airbyte? • Anything else I dont see? Did I oversee any documentation that already covering that topic? 🫣 Thanks in advance!

Loved the demo of the Custom Builder today. We upgraded our Airbyte instance to the latest release

0.44.4

but I am not seeing the “Builder” tab. What are we missing?

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

how to override the default airbyte username and password, looking to set a different password, currently the configuration api authentication is based on this username and password.