Hi everyone! Another lakeFS webinar is here!! Delivering high-quality data products requires strict testing of pipelines before deploying those into production. Today, to test using quality data, one either needs to use a subset of the data, or is forced to create multiple copies of the entire data. Testing against sample data is not good enough. The alternative, however, is costly and time consuming. We will demonstrate how to get the entire production data set with zero-copy. You will learn: 1. Set up your environment in under 5 minutes 2. Create multiple isolated testing environments without copying data 3. Easily run multiple test on your environment using git-like operations (such as commit, branch, revert, etc.) When: 10:00 AM PST, 10th Nov 2022 Where: Virtual - https://go.lakefs.io/3te313L

Hello Community! We have announced the schedule for Day 1 of move(data), the virtual conference celebrating data engineers' hard work and dedication worldwide! On December 7th come see leaders from Astronomer, Atlan, Dremio, Monte Carlo, Prefect, and Starburst speak! Come claim your ticket now at movedata.airbyte.com!

Hey everyone! I’m working on a blog series to simplify the semantic layer (recently launched at Coalesce 2022!) and help companies make practical decisions regarding the implementation of this new technology. Currently, the semantic layer suffers from the problem of obscurity. While there’s a lot of marketing and conceptual content available around it, there’s not enough content that one can use to decide whether it makes sense for their organisation. Read on the first post of the series: Is the semantic layer suitable for you? where I provide a basic checklist of answers to make your decision making journey easy! Let me know what you think! 🙌🏽

Happy Friday folks! At Houseware, we’re proud to announce our SOC2 Type II certification. This certification goes a long way in reciprocating the trust that customers place in us by using our product, and affirms our commitment to the highest standards of security for our customers’ data. Read on to find out more about our journey to SOC2 Type II Compliance.

Hey folks, happy Friday 👋🏽 ! We’ve published a new blog post on How to Implement the Semantic Layer. In the blog post, we walk through a step-by-step process for organisations to evaluate their data needs and go about building a semantic layer from scratch! We’ve also included a small framework at the end to help you decide whether to build or buy! This is a follow up to our last post where we see how the Semantic Layer can add value to your organisation. If you haven’t, do check it out! Let us know what you think! 🙌🏽

Many of us stroll the worlds trying to understand – How can we optimize our data systems to achieve more? More deliverables, more volume, more valuable INSIGHTS. One valuable practice is CI/CD for data systems. 🤔 What is CI/CD for data? CI/CD stands for continuous integration and continuous delivery, and it is a software development practice that involves regularly merging code changes into a central repository, building and testing the code automatically, and deploying it to production. In the context of data, CI/CD can be used to automate the process of integrating, validating, and deploying data pipelines and models. This can help to ensure that data is consistently processed and made available in a timely and reliable manner, enabling data-driven applications and services to function smoothly and effectively. ✅ How do I implement CI/CD for data? To achieve CI/CD for data, all you have to do is lakeFS your data. Yes, just lakeFS it! Curious to learn more? Join our O'Reilly course on CI/CD for data.

wow i kinda like this data scaling chart, i’ve never seen it laid out like this

👋 i'm wrapping up a blogpost on Data Mastodon - is anyone else on mastodon already? wanted to add it to the end of our list so that people can find you

This message was deleted.

This message was deleted.

This message was deleted.

@Mitali Narula has left the channel

This channel was disconnected because it was shared with DecisionTree Analytics & Services but everyone from DecisionTree Analytics & Services has left the channel.

This message was deleted.

This message was deleted.

Thanks for having me on the call @Chris Rose (Airbyte) and team! For those on the call, free to Slack me directly if there were questions we didn’t get to, or you want to chat Airflow, Airbyte, DBT!

has renamed the channel from "random-news-and-events" to "help-infrastructure-deployment"

set the channel topic: Post your issue while using Airbyte Platform during deployment, upgrading.

Was directed here from #C021JANJ6TY We just migrated one of our Airbyte instances from a standalone server to Kubernetes, and recreated all the connections. All but one of the connections is working after the migration. When we click Sync Now on the connection having problems, we get the error

Failed to start sync: The connection manager workflow for connection bc2a79bb-bb0d-42b4-9492-8dcdbf5e7d4e is deleted, so no further operations cannot be performed on it.

Hi all! We wanted to know what OSS deployment is the most stable and close to Airbyte Cloud there is. We have tried and stuck with docker deployment on GCP VMs so far and have tried helm deployment into GKE half a year ago

Hi all, is it possible to config Airbyte OSS to use an external instance of temporal? I already have temporal up it production and would love to config Airbyte to use that. rather than running two instances of temporal.

Hi, we have some problems with running airbyte on k8s. We use karpenter for scaling and I noticed that when some workloads are run into different machines (e.g. source on node a, destination on next node) sync failed, if everything run on one single node everything works fine. I got an error on when normalization is executed.

2023-04-27 07:05:04 normalization > 21 of 68 OK created view model ***.***............................................. [SUCCESS 1 in 3.87s]
2023-04-27 07:05:04 normalization > 20 of 68 OK created view model _AIRBYTE_AIRBYTE_SCHEMA.***................................................. [SUCCESS 1 in 4.04s]
2023-04-27 07:05:04 normalization > 25 of 68 START incremental model ***.***........................................................... [RUN]
2023-04-27 07:05:04 normalization > 26 of 68 START incremental model ***.**........................................................... [RUN]
2023-04-27 07:05:04 normalization > 27 of 68 START table model ***.***................................................... [RUN]
2023-04-27 07:05:09 normalization > 27 of 68 OK created table model ***.***.............................................. [SUCCESS 1 in 4.49s]
2023-04-27 07:05:09 normalization > 28 of 68 START incremental model ***.***......................................................... [RUN]
2023-04-27 07:05:13 INFO i.a.w.p.KubePodProcess(close):760 - (pod: airbyte / normalization-snowflake-normalize-19-1-nlpfa) - Closed all resources for pod
2023-04-27 07:05:13 INFO i.a.w.n.DefaultNormalizationRunner(close):194 - Terminating normalization process...
2023-04-27 07:05:13 ERROR i.a.w.g.DefaultNormalizationWorker(run):86 - Normalization failed for job 19.
io.airbyte.workers.exception.WorkerException: Normalization process did not terminate normally (exit code: 137)
	at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:205) ~[io.airbyte-airbyte-commons-worker-0.43.1.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:84) ~[io.airbyte-airbyte-commons-worker-0.43.1.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:37) ~[io.airbyte-airbyte-commons-worker-0.43.1.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$6(TemporalAttemptExecution.java:202) ~[io.airbyte-airbyte-workers-0.43.1.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

I’ve noticed that process was killed

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  39s   default-scheduler  Successfully assigned airbyte/normalization-snowflake-normalize-17-1-qhcws to ip-10-197-18-83.eu-west-1.compute.internal
  Normal  Pulled     39s   kubelet            Container image "busybox:1.28" already present on machine
  Normal  Created    39s   kubelet            Created container init
  Normal  Started    39s   kubelet            Started container init
  Normal  Pulling    36s   kubelet            Pulling image "airbyte/normalization-snowflake:0.4.0"
  Normal  Pulled     21s   kubelet            Successfully pulled image "airbyte/normalization-snowflake:0.4.0" in 15.303836555s
  Normal  Created    21s   kubelet            Created container main
  Normal  Started    21s   kubelet            Started container main
  Normal  Pulled     21s   kubelet            Container image "alpine/socat:1.7.4.3-r0" already present on machine
  Normal  Created    21s   kubelet            Created container relay-stdout
  Normal  Started    21s   kubelet            Started container relay-stdout
  Normal  Pulled     21s   kubelet            Container image "alpine/socat:1.7.4.3-r0" already present on machine
  Normal  Created    21s   kubelet            Created container relay-stderr
  Normal  Started    21s   kubelet            Started container relay-stderr
  Normal  Pulled     21s   kubelet            Container image "curlimages/curl:7.83.1" already present on machine
  Normal  Created    21s   kubelet            Created container call-heartbeat-server
  Normal  Started    20s   kubelet            Started container call-heartbeat-server
  Normal  Killing    16s   kubelet            Stopping container main
  Normal  Killing    16s   kubelet            Stopping container call-heartbeat-server
  Normal  Killing    16s   kubelet            Stopping container relay-stdout
  Normal  Killing    16s   kubelet            Stopping container relay-stderr

Killing process happened immediately when call-heartbeat-server started so I looked into entrypoint of this container:

trap "touch /termination/check" EXIT
      (set -e; while true; do curl -s 10.197.19.169:9000 &> /dev/null; sleep 1; done) &
      CHILD_PID=$!
      (while true; do if [ -f /termination/main ]; then kill $CHILD_PID; exit 0; fi; sleep 1; done) &
      wait $CHILD_PID
      EXIT_CODE=$?

      if [ -f /termination/main ]
      then
        exit 0
      else
        exit $EXIT_CODE
      fi

I confirmed that there is communication between pods in different nodes. Do you have any ideas what I can do with this issue? or maybe this is some limitation and whole sync process should be done on single node.

Hi All, We have problem deploying airbyte on EKS cluster using helm chart, We are using external minio and postgress in our setup, after deploying when I am trying to create any source connection in UI I get the following error on checking the worker and server pod logs I could see the following detailed error message, even though I am specifying minio bucket to use why is it connecting to S3, Any help will be really appreciated, let me know if I need to share values.yaml file used for deployment

Collecting content into /tmp/toBePublished6209292906792156831.tmp before uploading.
Cannot start publish with com.van.logging.aws.S3PublishHelper@703961dc due to error: Cannot start publishing: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null)
Publishing to S3 (bucket=sample-bucket; key=job-logging/workspace/66cd15e7-0f0c-4a7c-89be-79863ba48b6b/0/logs.log/20230427075426_int-qa-airbyte-worker-59ff685c8f-s76xt_dcce21db15244bf8a191652ceb5710f5):
java.lang.RuntimeException: Cannot publish to S3: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null)

Hello. When I am running run-ab-platform.sh script I am getting an error: “ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type [io.airbyte.config.persistence.ConfigRepository] airbyte-worker | airbyte-worker | Message: /flags (Operation not permitted)” …. “airbyte-server exited with code 1" Any clues what is going on and how to fix it?

Hi there. I use Airbyte to sync tables from Postgres to Snowflake. Is there an easy way to make Airbyte tables (e.g. _AIRBYTE_RAW_*) as transient tables, instead of permanent tables?

This message was deleted.

I set up sending metrics to Datadog but only see less than 10 metrics. Is that all metrics Airbyte provides? I look into this doc https://docs.airbyte.com/operator-guides/collecting-metrics/ which says “Visit OssMetricsRegistry.java to get a complete list of metrics Airbyte is sending.“, but the link leads to an non-existing file.

This message was deleted.

Hi, I have deployed Airbyte on AWS, where can I change the username and password? local .env file?

Hey folks, I have set up Airbyte on an EC2 instance and I want to set up DNS on it. For that, I need to change the port for the webapp to 443. Does anyone know how to do this? I tried changing in .env file (WEBAPP_URL), but that did not work for me.