Airbyte #ask-ai

xi-chen.qi

06/05/2023, 9:47 AM

Hi,Teams.I am using OpenTelemetry to monitor the Airbyte On Helm (EKS，version:0.45.12) cluster, and the following error is reported during the deployment process:

Copy code

Jun 05, 2023 9:17:32 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument num_orphan_running_jobs has recorded multiple values for the same attributes.
Jun 05, 2023 9:17:32 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument num_pending_jobs has recorded multiple values for the same attributes.
Jun 05, 2023 9:17:32 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument num_running_jobs has recorded multiple values for the same attributes.
Jun 05, 2023 9:17:32 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument oldest_running_job_age_secs has recorded multiple values for the same attributes.
Jun 05, 2023 9:17:32 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument oldest_pending_job_age_secs has recorded multiple values for the same attributes.
Jun 05, 2023 9:17:32 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
SEVERE: Failed to export metrics. The request could not be executed. Full error message: Failed to connect to otel-collector/172.20.121.101:4317

otel-collector yaml file：

Copy code

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: localhost:4317
    processors:
      batch:
      memory_limiter:
        limit_mib: 1500
        spike_limit_mib: 512
        check_interval: 5s
    exporters:
      logging:
        loglevel: debug
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]
      telemetry:
        metrics:
          address: localhost:8888

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    componet: otel-collector
    app: opentelemetry
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-collector
  replicas: 1
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-collector
    spec:
      containers:
        - command:
            - "/otelcol"
            - "--config=/conf/otel-collector-config.yaml"
          image: "otel/opentelemetry-collector:latest"
          name: otel-collector
          ports:
            - containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
            - containerPort: 8888 # Port for Prometheus instance
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - configMap:
            name: otel-collector-conf
            items:
              - key: otel-collector-config
                path: otel-collector-config.yaml
          name: config

---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  ports:
    - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
      port: 4317
      protocol: TCP
      targetPort: 4317
    - name: metrics
      port: 8888
  selector:
    component: otel-collector

Pierre CORBEL

06/05/2023, 11:58 AM

Release cycle

Hello, I saw that since last week, the number of release on https://github.com/airbytehq/airbyte-platform/releases have exploded. Did you move to an automatic nightly release cycle? Or is it just temporary and nothing official?

Octavia Squidington III

06/05/2023, 7:45 PM

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

Ryan Lewon

06/06/2023, 3:29 AM

Hey everyone, not so much a connector or infra issue but say you have a password from a vendor - and you cannot find it in our vault, I found airbyte.secrets in the postgres db but the

airbute_workspace_${workspace_id}_secret_${whatisthis_id}_v1

does not align with anything remotely to the connectors name. Is there any documentation from the infra side that shows what that "whatisthis_id" and where it links back to a connector in postgres?

Andreas

06/06/2023, 7:38 AM

I'm getting the following error when trying to upgrade: "Message: Instantiation of [simple type, class io.airbyte.featureflag.ConfigFileFlag] value failed for JSON property serve due to missing (therefore NULL) value for creator parameter serve which is a non-nullable type" Does anyone have an idea on how to solve this?

Octavia Squidington III

06/06/2023, 1:45 PM

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

Slackbot

06/06/2023, 5:04 PM

This message was deleted.

karl

06/06/2023, 5:10 PM

Does anyone have a best practice regarding backing up and restoring credentials? For instance, we’d like to be able to do the following: • backup: regularly back up sources, destinations & connections in a git repo • restore: in the event that our airbyte installation gets messed up somehow, be able to restore all the sources, destinations & connections from the yaml files The backup step can be done pretty easily by exporting these things to yaml files and saving the yaml files in the repo. But the restore step is a little more tedious / painful because all the credentials would have to be added by hand, locally. And I don’t know if I want to write a script that somehow pulls all the credentials from <somewhere> because just having a script like that could be a security vulnerability?

Lucas Abreu

06/06/2023, 8:07 PM

Hi everyone I'm still reading more on the docs to know if AirByte suits our needs, so sorry if any question is really newbie. Also Posting here as I'm not aware a better channel. Reading the doc using Postgres CDC, it states that once a schema change is made in the source database, one needs to restart the sync. Questions: • By sync restart: does Airbyte start the replication from zero ? I.e. another consistent snapshot from the databsae ? If it does start from scratch, do we have benchmark on latency ? • Am I able to restart only the schema modified table or is that connection level, i.e., apply to all tables in the connection ?

Kartik

06/06/2023, 8:56 PM

hey team, im looking for a FEDRAMP compliant solution, how can I make airbyte fedramp complaint?

Alex Faria

06/07/2023, 2:56 PM

Hello, I'm evaluating the use of airbyte and in the comparison of versions, I understood that the SSH tunnels functionality only exists in the paid version. I would like to explore this point further, in the case of the free function does it mean that the connection between airbyte and my database will not be done through an encrypted tunnel?

Kevin Phan

06/07/2023, 4:37 PM

hey all, i am deploying airbyte via flux and this is the helmrepo defined in my airbyte.yml file:

Copy code

apiVersion: <http://source.toolkit.fluxcd.io/v1beta2|source.toolkit.fluxcd.io/v1beta2>
kind: HelmRepository
metadata:
  name: airbyte
  namespace: flux-system
spec:
  interval: 1h0m0s
  url: <https://airbytehq.github.io/helm-charts>

deployment worked but i am still stuck at 0.40.0-alpha
. Do i need to be more specific with this? i am trying to upgrade airbyte to the most recent version to upgrade the jira connector.

Thiago Guimarães

06/07/2023, 8:03 PM

Im using the helm kubernetes deployment. The normalization jobs arent respecting my JOB_KUBE_TOLERATIONS and my node selector for the pool. All other job pods are respecting my node selector and tolerations

Thiago Guimarães

06/07/2023, 9:03 PM

@kapa.ai where can i report a bug in the kubernetes deployment helm chart ?

Taylor

06/07/2023, 11:15 PM

I'm working on deploying airbyte to an EC2 instance, but I'm having some problems with installation. I've verified that docker compose is installed but when I start airbyte it throws a warning saying docker compose v2 is not found even though I have verified the installation

Soshi Nakachi仲地早司

06/08/2023, 2:40 AM

I am currently deploying Airbyte with kubernetes. I have been using kustomize, but it has been deprecated and I am considering moving to using helm. So, is it possible to define my own additional resources? For example, Ingress of airbyte-webapp or Service Account. We have already defined them in kustomize and are operating with them, but is it possible to do the same with helm?

Sivakumar Ramaswamy

06/08/2023, 4:38 AM

Hi Team, i am applying service mesh on Airbyte pod as per company security policy however, it seems, airbyte does not support linkerd out of the box. is anyone implemented linkerd on airbyte ? Worker is throwing below error and also, i have reported in github https://github.com/airbytehq/airbyte/issues/27102

Copy code

2023-06-08 04:04:42 WARN i.t.i.w.Poller$PollerUncaughtExceptionHandler(logPollErrors):300 - Failure in poller thread Workflow Poller taskQueue="CHECK_CONNECTION", namespace="default": 1
io.grpc.StatusRuntimeException: UNAVAILABLE: Subchannel shutdown invoked
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:3750) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.WorkflowPollTask.doPoll(WorkflowPollTask.java:140) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:122) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:43) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:274) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:237) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Sivakumar Ramaswamy

06/08/2023, 6:35 AM

Hi Team, How to configure autoscale of worker based on total number of jobs in k8s ? I could not find any documentation on how to do ? could you provide pointer on this ?

Yogic Wahyu

06/08/2023, 6:43 AM

Hi Team, My team just found that Airbyte k8s (0.40.28, GKE) got frequent errors within 2-3 days of running (If it’s happen then all jobs will be stuck or error and we need to restart/re-install k8s). I know that several threads has discussed about this, https://airbytehq.slack.com/archives/C021JANJ6TY/p1677680873304649 https://airbytehq.slack.com/archives/C01AHCD885S/p1684769240244449 *Snippet of logs

Copy code

io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed

io.airbyte.workers.exception.WorkerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "io.airbyte.workers.process.KubePortManagerSingleton.take()" is null

Easy solution, we will upgrade airbyte to latest version. We are not confident enough that this will solve the problem. Do you guys have any workaround or clues about this behavior/things?

ratibor78

06/08/2023, 12:50 PM

Hi Team, is it possible to use external PostgreSQL database for the K8s Helm airbyte deployment ? I try to use values.yaml file with the hostname and credentials but I have a filling that airbyte still use internal DB that it creates. Thanks

Octavia Squidington III

06/08/2023, 1:45 PM

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

ratibor78

06/08/2023, 2:03 PM

Is it also possible to store the all logs for airbyte in external storage like S3 bucket for k8s airbyte deployment with Helm? What values I need to add to values.yaml ? Thx

ratibor78

06/08/2023, 2:10 PM

How can I migrate my existing jobs and all things from existing Airbyte on EC2 instance to my new installed airbyte on k8s with external PostgreSQL DB ?

ratibor78

06/08/2023, 2:53 PM

One more question if your pleasure, why the Airbyte S3 setup in values.yaml have the bucketRegion: "" ? for example my S3 buckets all in Global region but not in specific one. Thx

ratibor78

06/08/2023, 3:47 PM

Is it possible to send Airbyte logs form k8s installation to AWS CloudWatch ? Thx

ratibor78

06/08/2023, 4:17 PM

Is it also possible to store the all logs for airbyte in external storage like S3 bucket for k8s airbyte deployment with Helm? What values I need to add to values.yaml ? Thx And where I need to put my IAM user key and secret key ?

MOHAMMED ALQURASHI

06/09/2023, 6:22 AM

Hi… how can I modify the docker compose file to add additional workers ?

Ignacio Contreras

06/09/2023, 11:47 AM

Hi! I have Airbyte in a VM the images are using lots of space. Is there a way to remove old image versions? Besides deleting one by one? Here is an example of all postgres images. Currently, I’m only using airbyte/source-postgres:2.0.33 so I would like to delete the remaining ones.

Omar Allouache

06/09/2023, 4:20 PM

Hello, I am trying to install octavia-cli inside a docker container (docker in docker), anyone tried to do it ? I tried with docker:stable-dind container but it do not have bash, curl etc ...

Octavia Squidington III

06/09/2023, 7:45 PM

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!