I'm very confused

https://github.com/twelve-factor 12-factor is changing... but it's going to evolve as an open source project? Still not quite sure what this means, but it was just announced at KubeCon.

1731613899629.png

Tangentially related to Kedro, but a while ago I saw a post (maybe Linkedin) about how people are realizing that distributed data processing tools (e.g. PySpark, maybe?) aren't actually necessary And that DuckDB (I think) was enough for 95% of applications since people are usually under 1TB scale for most operations. Anyone know where I could read more about something like this?

Great post by Armin fundamentally on the topic about the complexity risks of giving developers too much freedom https://lucumr.pocoo.org/2024/11/26/python-packaging-metadata/

any good black friday deals ? 😄

✈️ Thrilled to share the incredible progress of Ai.rplane at PhysicsX! This project combines cutting-edge experience design, AI physics simulation, and world-class engineering to push the boundaries of what’s possible in generative AI and aviation. The models powering Ai.rplane were created using Kedro pipelines ❤️ Give it a try and let me know what you think!

I wanted to solve Day 5 of Advent of Code with Kedro (since it can be modeled as a topological sorting problem) but 🥲

Hi all, Looking for code analysis tools that can automate as much checks as possible, such as typos, code format, docs language, etc. With the goal to have as strict toolchain as possible. This is what I'm using now:

dev = [
    "codespell~=2.3.0",
    "import-linter==2.1",
    "mypy~=1.13.0",
    "pre-commit~=4.0.0",
    "pytest~=8.3.3",
    "ruff==0.7.1",
]

Are there any other tools you can recommend? (I did ask ChatGPT but the suggestions aren't great). Assuming that things like

isort

,

black

,

flake8

etc. are all covered by

ruff

.

those of you learning Spanish, I hope you appreciate this piece of beauty in the Azure docs

TIL: YAML merge keys (

<<:

) were never part of the spec 😳 https://ktomk.github.io/writing/yaml-anchor-alias-and-merge-key.html

This is very cool 🦆 https://bsky.app/profile/hamilton.bsky.social/post/3lerfa3ljfc2j

dbt just acquired one of its most interesting competitors, meaning it's now a fight between them and Tobiko

will be verrrryyy interesting if the sdf vision gets wrapped into dbt or it's just a case of catch and kill

I'm curious, is anyone here using EdgeDB? I was watching an interview with the founder yesterday, so wanted to understand how popular it is.

I just started using marimo for real today and it's beautiful https://marimo.io

Interesting PEP maybe the death of Jinja? https://peps.python.org/pep-0750/

speaking of PEPs: https://peps.python.org/pep-0771/ this would be very relevant for Kedro

A bit of shameless self-promotion 🙈 I was interviewed for the AI Insights Podcast and talked about my career so far, Kedro K, working on and maintaining an open-source project and the future of AI. 🎧 Give the episode a listen on Spotify: https://open.spotify.com/episode/17WHl9Yj6qgr44R4OskxUx?si=7xqpbvjeQW-yccjllk2OLA

I've taken notes on the tutorial series that was produced last year. The notes are on my blog. Feel free to point out any errors or typos. There undoubtably are some. Notes on Kedro Tutorial Videos

Life-changing dev tool I learned about recently: https://github.com/nektos/act Been using it on the https://github.com/dagster-io/community-integrations repo; works like a charm for almost everything there.

Hi @Juan Luis (and everyone) here you see two Kedro users (a beginner and an experienced one) meeting by chance at an Open Source Saturday meetup in Turin :). w @Giovanni Luddeni

I am conflicted on naming conventions for columns. Do I use spaces or underscores to separate words? • Underscores make it easier to select/edit the entire column name. • Spaces enable narrower columns due to word wrapping. I prefer longer column names (fewer abbreviations) to improve clarity. What are your thoughts?

I was suddenly reminded by the geospatial hackathon idea in #C03RKAQ0MGQ... If you do geospatial work and haven't tried DuckDB for geospatial data, it's apparently really good. I'm no geospatial expert, but last year we got a chance to see how one expert replaced complex Dask-based workflows with DuckDB and Ibis. https://docs.fused.io/blog/how-digitaltwinsim-models-wireless-networks-with-duckdb-ibis-and-fused/ showcases some of this publicly.

💡Cool insight on where Kedro is still introducing performance overhead from a use case I'm working on In almost all cases, either I/O or node execution time will outweigh this overhead by a lot, making it irrelevant but I'm sharing it anyways 😄 In my case, I am running ~800 dynamically created small nodes (the nodes are all interdependent in a very complex relationship graph so I rely on Kedro's orchestration without the need for me to codify the execution order 🙏) Initially my pipeline took 1min40, and the following two changes reduced it to 8sec: • Disabling pluggy tracing (see issue) • Using "list" inputs instead of "dict" inputs in

node

Post