Last Friday I spoke at the OpenText conference in Malta. As usual, Meltano was part of my stack. Lots of people were interested in Meltano afterwards, for example people from Estonia who are working on further developing the already advanced digitalisation of government. I invited them to join this community, hopefully they will come 😉

Hello Meltano Community! melty We recently published an article on creating a 'Private Meltano Hub' - essentially a way to run a secure, enterprise-grade Meltano Hub. Imagine a place where you could easily publish your own suite of Meltano plugins for your organization! I'd love to hear your feedback on the idea. Happy Friday! https://medium.com/@rfrankel_8960/a-private-meltano-hub-for-enterprise-users-b23c324632d0

Recent Motherduck annoucement: https://motherduck.com/blog/announcing-motherduck-duckdb-in-the-cloud/ There are many compatible components (dbt, Airflow, ...), but not Meltano. You should be there 😉 With Motherduck it is no longer about a poor-man solution, IMHO, Motherduck can be the DWH on top of which analytics platforms can directly run.

... https://twitter.com/TarasNovak/status/1673788758846734337

Our upcoming release (1.4) will feature some major upgrades in Dagster’s support of dbt. We will host a special event on Aug 2nd to showcase these new capabilities. You can sign up for a reminder for the event and dial in details here: https://dagster.io/events/dbt-event-aug-2023

As many of you have noticed ClickHouse and Kubernetes work great together. It’s easy to stand up toy applications, but what about building an entire analytic service based on ClickHouse? This coming Thursday Robert will show you how to build the full stack using Kubernetes, ArgoCD, and open source software. There’s even a GitHub project under Apache 2.0 license with the code to do it yourself. Please join us to learn more: https://hubs.la/Q01WsbS_0

If you use dbt in your pipelines and you are frustrated by how slow dbt docs runs, check out this post by the Dagster team: Speeding up the dbt™ docs by 20x with React Server Components https://dagster.io/blog/dbt-docs-on-react

For anyone interested in integrating Arrow in Python/Java/Kotlin, my colleague wrote this nice article: https://medium.com/gooddata-developers/polyglot-apache-arrow-java-and-python-perspective-bf2ce020e27d

in case some of you are interested to try new pro data tools for vscode ide, you can get them today for $0.66. See new changelog info post for details: https://twitter.com/TarasNovak/status/1687098946131214337

Dagster is going to London next week - if you are in town, come join us in Hammersmith in coordination with our friends at Rittman Analytics: https://www.eventbrite.com/e/pre-big-data-london-kickoff-meetup-tickets-700781495317

Hi, we are going to do the third meetup in our Prague offices next Tuesday, Sep 19th, at 6pm CET. This time we will also stream it live with Zoom. Me(GoodData), Petr Chocholous(Keboola) and Martin Cmejrek(Mama AI) will talk about real-life implementations of ELT and Business Intelligence solutions powered by generative AI. https://www.gooddata.com/resources/goodmeetup-three/

Patrik Braborec, my colleague, just published a very nice article about how we leverage LLM/generative AI for analytics use cases. It is about what we are currently cooking and want to ship to production. Feedback from you is exactly what we want! https://medium.com/gooddata-developers/how-to-build-data-analytics-using-llms-in-under-5-minutes-dd6984be2432

The Dagster team will be hosting our Fall Launch Week from October 6th to the 13th. Each day, we will announce and showcase new features and capabilities on the Dagster platform. The theme of Launch Week is “*Escaping the Modern Data Trap*.” Why? The advent of the Modern Data Stack revolutionized the domain of data engineering. But over time, it drifted from its original premise. It became fragmented, overcomplicated, and expensive to run. Join members of the Dagster Labs team as we share our vision of how to escape this Modern Data Trap and get back to the initial intent: to deliver a developer experience that is unified, streamlined, collaborative, and where you are in control of costs. We hope you will join us for this discussion and new updates: https://dagster.io/events/dagster-launch-week-oct-2023

We at Timeplus has open-sourced Proton, https://github.com/timeplus-io/proton, a streaming database written in C++, unifying streaming and historical data processing in a single binary, powered by ClickHouse. We also provide a serverless offering in the cloud, as a go-to analytic platforms for almost everything. Yes, we sent all kind of data to our Timeplus, without using any other databases(no snowflake, RDS, PG, mysql) A meltnao target to Timeplus is also built and published to meltano hub.

If you thought fuzzymatching was only for customers or suppliers, take a look at how Zingg can help with image matching! Written by Databricks, this ready made accelerator can help you match textual and image data, all in one go. I can think of use cases like competitive pricing and KYC. Would love to hear how you use it and your feedback. https://www.databricks.com/blog/using-images-and-metadata-product-fuzzy-matching-zingg

Convergence of Business Intelligence and (even advanced) Machine Learning? We implemented the first piece of our vision of how the convergence could look like. We allow business users to forecast in dashboards with one click, and data science persona to open the corresponding notebook to fine-tune it. My colleague Stepan Machovsky wrote a very nice article about it! https://medium.com/gooddata-developers/machine-learning-in-dashboards-b13d9ba7855a

Dagster 1.5 shipped this week - you can read about the major updates here: https://dagster.io/blog/dagster-1-5-how-will-i-know This sets the stage for next week’s Launch Week - be sure to join us, starting on Friday 6th, as we dive deeper into the new capabilities of Dagster: https://dagster.io/events/dagster-launch-week-oct-2023

Hi everyone 👋! The Open Source Analytics Conference is happening on Dec 12-14. It is FREE and VIRTUAL. Learn from top industry experts and connect with the community. See what Kafka, Airflow, Superset, Meltano, ClickHouse, DuckDB and your other favorite projects are up to. We have an amazing lineup of speakers, check it out here: https://hubs.la/Q028Sx0h0

I decided we had to join the flock but I didn't want to rush it. So I asked myself: is DuckDB, or MotherDuck, proDUCKtion-ready for BI/analytics use cases? You can find a thorough(and positive) answer in my article. Moreover, you will understand what it means to integrate a new database into a BI platform. Huge thanks to the MotherDuck community for all the help, they rock! https://medium.com/gooddata-developers/is-motherduck-producktion-ready-a3a0347715c5

We are happy to announce a new series of Data Engineering with Dagster events: In these five sessions, we will cover specific aspects of the Dagster Framework. https://dagster.io/events/dagster-deep-dives-feb-2024

And we will be hosting our second Dagster Deep Dive event later today: Session 2: Configuration & Resources Tuesday, February 20th with Colton Padden 8:30 AM PST / 11:30 AM EST / 4:30 GMT / 5:30 CET Are you done with Airflow? If you’ve ever had to install Kubernetes locally just to test a simple pipeline, or have resorted to the push-to-prod-and-pray method, it’s time to take a look at how Dagster’s configuration and resource systems allow you to develop locally and ship confidently. Add it to your calendar here: https://www.addevent.com/event/Gy20086226

@Taylor Murphy (Arch) quickly wrote up a guide on credentials management for data consultancies, great read! https://arch.dev/blog/why-your-data-consultancy-might-be-one-password-away-from-disaster-and-7-ways-to-fix-it/

I wrote up a quick opinion on why data consulting clients are often resistant to data literacy and how to tackle that challenge! Take a look here: https://arch.dev/blog/why-your-clients-are-resistant-to-data-literacy/

In NYC Monday? Join us and LakeFS for a Monday Morning MeetUp in Midtown. Dagster Labs founder & CTO Nick Schrock will be joining Oz Katz, Co-Founder & CTO of lakeFS share their expert knowledge on how to troubleshoot and reproduce data to ensure only quality data is exposed to production. Details: March 11, 2024 930 1130 Microsoft Reactor Times Square. Register here: https://lakefs.io/event/meetup-dagster-lakefs-240311/

melty bouncyI've been quite productive lately, thinking a lot about productizing data services. Here's an opinion on how to fight the fears common to the productization journey for data consultancies. https://arch.dev/blog/productizing-data-services-removing-fears-with-the-leap-framework/

On April 17th we will be unveiling Dagster+, the next iteration of Dagster Cloud. Find out more about the launch here. Prior to this on April 12th we will be hosting two happy hours - one in NYC and one in SF. If you would like to come meet the Dagster Labs team, sign up here. Spots are limited but we will do our best to accommodate everybody. We hope to see you there!

Day1 Summit : 90 day blueprint to succeed in new data role Starting a new data role can be daunting. Where do you begin? How do you make an impact? How do you earn trust and credibility? All the questions flood in but there's no one to guide. With Day1 Summit, we want to equip you with a 90-day game plan to nail your new role! Date: 29th May Time: 11:30 AM EST Location: Virtual Check it out here: https://bit.ly/3wJk3ME

Hey folks because so many of you (>2/3) use LLMs for coding, I started exploring it so we can create some docs/tutorials soon Here's a video how I converted an airbyte yml source to dlthub rest api (holy moly this works too well)