I’d really like to know/read more about how advanced kedro users use kedro. Beyond the first pipeline being set up: • how do you add new features to existing pipelines, • how do you do dev -> prod promotion, • do you use notebooks to iterate on new ideas then merge or develop directly in scripts? • How do you deal with pipelines that do retraining or incorporate model drift/data drift scenarios? • Do you use the kedro cli or configure more advanced runs via the python sdk?

Someone will reply to you shortly. In the meantime, this might help:

These questions arise from experience and frustrations after using kedro for several months. Namely in my team a training Jupyter notebook has been the basis for training new models for each client we onboard (demand forecasting) and we use kedro pipelines for predictions and refitting mainly. The reason Jupyter was chosen for the training of models rather than kedro directly is to have more interactivity with the idea of migrating the core/production training notebook into a kedro pipeline at some point. I am currently trying to steer my team away from using notebooks in this fashion and just go pipeline or script native but I’ve found significant push back, I wonder how others approach these issues

1. Depending on what you mean by features a. If you mean independent ML estimator variables, that's typically just a change in the YAML parameters file which adds new variables in) b. If you mean functionalities, that implies adding new nodes typically 2. I haven't been setting this up myself, but I think a common patters is to make both code and data have

dev

and

prod

versions (branches). 3. Never notebooks, only

.py

files, and I wouldn't call them scripts. I think of them more as like Python packages which contain data processing functions, and then Kedro is a very thin layer just to chain those functions together in particular order and pass data between them. 4. Trigger their runs manually every X days 5. Just the standard

run

CLI

That's really helpful @Yury Fedotov, thank you very much, I wonder what the core contributors think as well and whether it'd be worth having extended documentation or tutorials on this @datajoely @Juan Luis @Deepyaman Datta

I think it could be helpful, and a lot of it would be references back to/surfacing existing documentation:

• how do you add new features to existing pipelines,

This is really more about making sure your pipeline is well structured. https://web.archive.org/web/20250215132726/https://towardsdatascience.com/the-importance-of-layered-thinking-in-data-engineering-a09f685edc71/ (I don't know why I can't access the original article right now) by @datajoely is pretty representative of how used to lead data engineering teams where was adding hundreds of features; if you have a thought-out approach to storing features and a good way to combine them for consumption for modeling, the adding features bit should become easy. Our data science teams also used to have parametrized feature selection modules, etc.

• how do you do dev -> prod promotion,

https://docs.kedro.org/en/stable/configuration/configuration_basics.html#configuration-environments In terms of scaling compute/using the appropriate engines for dev and prod, I'm biased to https://kedro.org/blog/building-scalable-data-pipelines-with-kedro-and-ibis 😉

• do you use notebooks to iterate on new ideas then merge or develop directly in scripts?

Pretty commonly used to load data using the catalog into the notebook, and then do the prototyping (e.g. for a feature, or for modeling, if it needed some iteration), then move the stuff back into a pipeline quickly. https://docs.kedro.org/en/stable/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.html hits on some of this workflow (although at a much more basic level)?

• How do you deal with pipelines that do retraining or incorporate model drift/data drift scenarios?

What @Yury Fedotov said, and/or progress to orchestration with fancier tools

• Do you use the kedro cli or configure more advanced runs via the python sdk?

What @Yury Fedotov said again; not sure what configuring more advanced runs would be?

Thanks so much @Deepyaman Datta that's very enlightening, it seems like I need go do some studying on these articles Another question, how common is it in kedro to have a pipeline that trains a model or several models (as in my example of demand forecasting) and it's just parametrised to run for every new client? i.e. training a new model for a new client as a one off pipeline? Then have other pipelines that generate the predictions?

The other practice I've gotten into - my python business logic lives in an independently well tested package. The kedro code is an extremely simple declaration of flow. Even better if that package is in a internal artifact store.

Another question, how common is it in kedro to have a pipeline that trains a model or several models (as in my example of demand forecasting) and it's just parametrised to run for every new client? i.e. training a new model for a new client as a one off pipeline? Then have other pipelines that generate the predictions?

I think this is pretty standard? But it would be better if a DS or MLE who has done a lot of this work in Kedro more recently could answer; it's been 5 years since I've been working on demand forecasting models, and I don't remember the patterns 😅