Last week I was able to migrate (🔗 PR) my Meltano dogfood project from BigQuery to MotherDuck. All of these plugins already support it, which made it really easy to accomplish: • target-duckdb • dbt-duckdb • Evidence (shout out to @archie_sarre_wood for making a quick PR to add support in Evidence)

I made this free directory to find tools that support the integrations that you need! Meltano definitely shines in its coverage 💯 https://www.definite.app/connector-db

I've released a tap for the Salesloft platform written using the Meltano SDK 🥳: https://hub.meltano.com/extractors/tap-salesloft--markestey/ Previously the only other connector was using the Airbyte wrapper so now there is an option that is Singer native. Thanks to @pat_nadolny for adding it to the Meltano Hub!

Just merged an update to mdsinabox.com - now officially has evidence in as a util and some other goodness like an NFL 🏈 model if that floats your boat!

Hi everyone! We recently released a new Meltano utility plugin - dbt-artifacts. If you have a Meltano project already using dbt, you can use this tool to generate data lineage through a few Meltano commands. meltano Check out this article and video if you are interested: Data Lineage for DataOps

We just published a piece to the blog that blew me away: @pat_nadolny just did a deep dive on building LLM applications, and why Meltano (and other data eng tools) can be used to make it a lot easier. I just tried it out myself (adapted it to chat with my own blog) and it works like a charm, is super quick to modify, and really neat to use. For all the details, read this: https://meltano.com/blog/llm-apps-are-mostly-data-pipelines/

Look what I just found: https://sicarul.substack.com/p/meltano-the-universal-glue-for-your @pablo_seibelt.

Hey Meltano-ers, Some exciting stuff from the Evidence newsroom today. We’re launching Evidence Cloud - our deployment, hosting, and user auth service for Evidence projects. We think this is pretty cool because it means you can combine Meltano ELT, a MotherDuck DB, and an Evidence front end (with auth), for free. You can get an invite to Evidence Cloud here (if you dm me I may be able to fast track it). We’ve also raised a $2M seed round, led by A Capital, with participation by SV Angel, Y Combinator, Tristan Handy, Joe Morrisey, Tido Carriero and several other angels.

New Meltano Substack post! https://sicarul.substack.com/p/gotta-catch-em-all-pokemon-with-meltano

Sorry, will fix it

Hi everyone! We've made a guide on how we normally implement DataOps change promotion with GitHub, Meltano and Matatika meltano The gist of the idea might be helpful, wherever you run your own project. Check out this article and video if you are interested: How to Promote Change in your DataOps Project

Hi! Some new stuff from us at Storebrand: A new variant of

map-gpt-embeddings

, made for Azure OpenAI: https://github.com/storebrand/map-gpt-embeddings A new variant of

target-postgres

, designed for the postgres

pgvector

extension. https://github.com/storebrand/target-postgres. Writes arrays as the custom Vector datatype, intended to be used together with

map-gpt-embeddings

. Otherwise identical to Meltanolabs’ variant. Two unpedagogical blog posts from me about writing tests for meltano taps and targets, a byproduct of me improving my pytest skills: Part 1: https://radbrt.com/posts/pytest-singer-1/ Part 2: https://radbrt.com/posts/pytest-singer-2/

melty bouncyOur CEO @Douwe Maan (Arch) just wrote down why we're building what we're currently building - a multi-tenant data platform! Take a look if you're interested in that. https://arch.dev/blog/why-the-modern-data-stack-sucks-for-data-consultancies-looking-to-productize/

melty bouncy I've spent some time researching the data consulting business. Here's a first take on it... (not about Meltano!) It's all about trust! Data consulting clients need it, yet it's so easily broken with ... - data that's out of sync for a day - not getting the (right) results - not seeing the results (even if you get them) And so much more. Check out our complete article here: https://buff.ly/4bix0MY

Based on this SO question: https://stackoverflow.com/questions/28798014/is-there-a-way-to-automatically-and-programmatically-download-the-latest-ip-rang I've built a very simple Meltano SDK tap that goes to the Microsoft Download page "Azure IP Ranges and Service Tags – Public Cloud", finds the download url for a JSON file (which changes weekly) and extracts the IP Address ranges used by Microsoft Azure. tap: tap-msipranges (open to suggestions on a better name!) https://github.com/mjsqu/tap-msipranges Inspiration - Microsoft's own scripts: https://github.com/microsoft/mstic/blob/master/.script/get-msftpubliip-servicetags.py If nothing else, it's an example of dynamically obtaining the base_url for a json-based tap, if you need to get that url from another statically named page. Paired up with target-snowflake, our Snowflake engineers are going to use this data to set security rules:

For the last 18 months, I have been gradually expanding my end-to-end data pipeline. I decided to declare it version 1.0 and write an article about it. The demo included - I added a new source - Jira API - for which I created a Jireaucracy dashboard. https://medium.com/gooddata-developers/data-pipeline-as-code-journey-of-our-blueprint-99912b1485d2

Just finished another blog post in the series on data consultancies: https://arch.dev/blog/9-ways-to-sell-data-services-to-non-data-savvy-clients/

I have been toying with a new cli utility to automate some dbt development - with OpenAI, naturally, as that is how we do things nowadays. The tool parses the dbt manifest so that it can add necessary context to the tasks. It currently does the following: • Create documentation for a model • Create unit tests for a model (dbt 1.8 alpha is out!) given a description of what to test. • Generate a new model based on description and names of input models. • Alter an existing model • Explain a model • Interactively chat with a model • Fluff and rewrite a model The utility is cheesily named “dbtai”: https://github.com/radbrt/dbtai I am not able to test this with a real-world dbt project (company policies and whatnot), but if you want to give it a try I’d be happy to hear feedback. Also, someone has to have made something like this before, so feel free to post links.

hey! a friend i have been working on a tool that generates OpenAPI spec from documentation links and then generates airbyte connectors. perhaps it could be valuable to generate meltano/singer connectors from the OpenAPI spec? would love your thoughts and feedback! https://github.com/skyffel/airbyte-connector-generator-poc

👋 Hey everyone, coming to you from Fleetio with our brand new tap-fleetio We would love for any current (or future!) Fleetio customers to try this out! If you would like to try this out, send me a dm 👀 (edited)

Hey all, I just published an article about Meltano. Please share your opinion. https://open.substack.com/pub/datagibberish/p/meltano-review?r=odlo3&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Hi all. We write a small post on using meltano to load test data. Enjoy! https://www.matatika.com/your-guide-to-loading-quality-test-data/

We are moving quickly from naive solutions to production-ready AI use cases. These require advanced approaches such as RAG architecture. I PoCed it and wrote an article about it. And yes, I used DuckDB as a vector database 😉 https://medium.com/gooddata-developers/a-way-to-production-ready-ai-analytics-with-rag-0c71fc3b23e8

OSA CON 2024 Call for Papers just opened. Time to highlight your open source analytics project or initiative: https://sessionize.com/osa-con-2024

Locking 1062 plugin(s)...

We crossed 1000 plugins recently (in one of our two Meltano projects). Across both Meltano projects we’ve now run a total of 419k

meltano run

jobs, according to the meltano system database.

https://github.com/meltano/hub/pull/1808/files Hi folks, I just opened a PR for an Apache Iceberg tap I've started to work on, currently passing the generic extractor test suite so I think ready to be added into the catalog. Slack Conversation

We've published a blogpost about our modified tap for Pulumi, based on Meltano's: https://www.pulumi.com/blog/import-your-pulumi-cloud-data-with-tap-pulumi-cloud/

Hey everyone, I've published a blog post about how we implemented

BATCH

support for

tap-bigquery

to significantly improve performance for large volumes of data. Give it a read if you are interested in why/how we did it! 😉 https://medium.com/@rfrankel_8960/incredible-bigquery-extract-performance-3432b9cd83fe Here is an example Meltano project that uses this tap: https://github.com/Matatika/example-ga4-export

📣 Hello all! The Open Source Analytics Conference 2024 is next week! 🚀 Some really cool talks on databases, orchestration, Bi/visualization tools! • When: Nov 19-21 • Where: Online! • More info: osacon.io Hope to see you there!