Hi everyone! 👋 I’m currently exploring how to approach unit testing in Kedro, especially when working with large-scale data pipelines. I’d love to hear your thoughts on a few things: • Do you find unit tests valuable in the context of data pipelines? • How do you typically implement them in Kedro? • Given that data quality checks are often a key focus, how do you handle testing when the input datasets are huge? Creating dummy data for every scenario doesn’t always seem practical. Any tips, examples, or lessons learned would be greatly appreciated! Thanks in advance 🙏

welcome @Wejdan Bagais! have you seen this? https://docs.kedro.org/en/stable/tutorial/test_a_project.html

@Juan Luis Thanks for sharing the link! I did check out the tutorial—helpful for getting started with testing in Kedro. That said, I’m still a bit unsure how useful unit testing is in my case. I’m working with a pretty large dataset that flows through several layers, and creating dummy data for each one doesn’t feel very practical. I totally see the value of unit tests for utility functions, but most of my nodes are tied to very specific data types or structures. So, I’m wondering if it makes more sense to focus on testing the data itself (like schema checks or data quality rules) rather than the functions. I'm curious how others are handling this.

So there are a few things: • in my opinion your Kedro flow should be dumb • If possible all of your complex business logic should be imported from independently well tested packages and modules which do not couple with Kedros API (this is a good SOLID principle as well as giving you an off ramp) • For complex data testing, it should like you need som expectation tests with pandera to enforce schema and expectations about the data such as nullability etc

I usually do unit tests for complicated pieces of logic, extensively codify Pandera checks (I did a wrapper that can handle pandas and spark interchangeably), and I set up a data fabricator for e2e pipeline smoke testing.

I think in general, for things that out of your control, test data quality for sure (data are produced by other team, API etc). For code that you can control, function logic are equally important. Then you can start adding data quality check for important dataset (dataset that connect with a lot of upstream/downstream nodes). You'll most likely want to spread out your data quality test so they can work like checkpoints.