Kedro #questions

galenseilis

11/19/2025, 10:30 PM

Why doesn't Kedro session internally bootstrap itself when being called outside of jupyter or kedro run? https://docs.kedro.org/en/stable/api/framework/kedro.framework.session/#kedro.framework.session.session.KedroSession

Yufei Zheng

11/20/2025, 5:35 PM

Hi team, I am very new to Kedro/Pyspark, we have some UDF function defined within Kedro pipeline.. I am wondering do we have an example of building the Kedro dependencies using

kedro package

and pass these to

spark executor

, thanks! (Tried to run the package command but still hitting

no module named xxx

in spark executor)

➕ 1

Ming Fang

11/21/2025, 12:22 AM

Hi. I'm starting to learn Kedro using the quickstart tutorial here https://docs.kedro.org/en/stable/getting-started/install/#installation-prerequisites I was able to run these commands

Copy code

uvx kedro new --starter spaceflights-pandas --name spaceflights
cd spaceflights

But the next command

Copy code

uv run kedro run --pipeline __default__

resulted in these errors

Copy code

[11/21/25 00:21:49] INFO     Using 'conf/logging.yml' as logging configuration. You can change this by setting the KEDRO_LOGGING_CONFIG environment variable accordingly.          __init__.py:270
                    INFO     Kedro project spaceflights                                                                                                                             session.py:330
[11/21/25 00:21:51] INFO     Kedro is sending anonymous usage data with the sole purpose of improving the product. No personal data or IP addresses are stored on our side. To opt   plugin.py:243
                             out, set the `KEDRO_DISABLE_TELEMETRY` or `DO_NOT_TRACK` environment variables, or create a `.telemetry` file in the current working directory with the              
                             contents `consent: false`. To hide this message, explicitly grant or deny consent. Read more at                                                                      
                             <https://docs.kedro.org/en/stable/configuration/telemetry.html>                                                                                                        
                    WARNING  Workflow tracking is disabled during partial pipeline runs (executed using --from-nodes, --to-nodes, --tags, --pipeline, and more).                  run_hooks.py:135
                             `.viz/kedro_pipeline_events.json` will be created only during a full kedro run. See issue <https://github.com/kedro-org/kedro-viz/issues/2443> for                     
                             more details.                                                                                                                                                        
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/coder/spaceflights/.venv/lib/python3.13/site-packages/kedro/io/core.py:187 in from_config  │
│                                                                                                  │
│    184 │   │                                                                                     │
│    185 │   │   """                                                                               │
│    186 │   │   try:                                                                              │
│ ❱  187 │   │   │   class_obj, config = parse_dataset_definition(                                 │
│    188 │   │   │   │   config, load_version, save_version                                        │
│    189 │   │   │   )                                                                             │
│    190 │   │   except Exception as exc:                                                          │
│                                                                                                  │
│ /home/coder/spaceflights/.venv/lib/python3.13/site-packages/kedro/io/core.py:578 in              │
│ parse_dataset_definition                                                                         │
│                                                                                                  │
│    575 │   │   │   │   "related dependencies for the specific dataset group."                    │
│    576 │   │   │   )                                                                             │
│    577 │   │   │   default_error_msg = f"Class '{dataset_type}' not found, is this a typo?"      │
│ ❱  578 │   │   │   raise DatasetError(f"{error_msg if error_msg else default_error_msg}{hint}")  │
│    579 │                                                                                         │
│    580 │   if not class_obj:                                                                     │
│    581 │   │   class_obj = dataset_type                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
DatasetError: Dataset 'MatplotlibWriter' not found in 'matplotlib'. Make sure the dataset name is correct.
Hint: If you are trying to use a dataset from `kedro-datasets`, make sure that the package is installed in your current environment. You can do so by running `pip install kedro-datasets` or `pip
install kedro-datasets[<dataset-group>]` to install `kedro-datasets` along with related dependencies for the specific dataset group.

Jan

11/21/2025, 9:33 AM

Hello everyone, we used Kedro to build a pipeline that validates datasets. The pipeline has grown a lot and sometimes we need validations to run in a certain order. I know that Kedro decides order based on node inputs and outputs and in the past we created dummy datasets to create dependencies but this makes the code harder to read and maintain. We also use Apache Airflow and in airflow dependencies are defined explicitly between nodes by a custom operator: Node1 >> Node2 >> Node3 This sounds like a problem that must have been encountered before so I wanted to ask around whether there already is a plugin or extension that provides this functionality?

➕ 1

Prachee Choudhury

11/22/2025, 3:44 AM

Hi Kedro Team, I recall there was a Python library/plug-in for text animation/design which may or may not have been a Kedro resource , which was posted about in the Kedro Slack. I cannot remember the name of the resource. Thank you

Ahmed Etefy

11/22/2025, 8:58 PM

Hey Kedro Team, I am wondering what the solution for the following use case is I have some iceberg tables, I run my pipelines with, and then at a later point in time I'd like to run on different versions of those iceberg tables leveraging the iceberg versions in the catalog so its committed in code Is there any recommendation for how to address this? Also should I be adding to the catalog multiple entries with different versions to reflect the different run instances? Does Kedro support in some way maintaining historical versions?

Basem Khalaf

11/22/2025, 10:26 PM

Hi Kedro Team 🙌 I’m currently running an older Kedro project (version 0.19.3), which is causing compatibility issues with the latest Kedro-Viz I have installed (12.2.0). Could you please advise where I can find the compatibility matrix between Kedro versions, Kedro-Viz versions, and the corresponding supported Python versions? Many thanks—I truly appreciate your help in advance. Basem

Ahmed Etefy

11/23/2025, 9:07 PM

Hey team Is there a way to have pipeline specific parameters.yml, and spark.yml? I'd ideally like to colocate pipeline config in the same folder as the pipelines for easier collaboration and I would like pipeline runs to only load pipeline specific configurations I guess what I am looking for is "composable pipeline projects"

Gauthier Pierard

11/24/2025, 1:48 PM

hey, i have an

after_context_created

hook called

AzureSecretsHook

that saves some credentials in

context

. Can I use these

credentials

as node inputs?

Copy code

context.config_loader["credentials"] = {
            **context.config_loader["credentials"],
            **adls_creds,
        }
self.credentials = context.config_loader["credentials"]

so far only been able to use it by importing

AzureSecretsHook

and using

AzureSecretsHook.get_creds()

directly in the nodes

Copy code

@staticmethod
    def get_creds():
        return AzureSecretsHook.credentials

Jonghyun Yun

11/25/2025, 4:31 PM

Hi Team, I have written Kedro pipelines for data processing, model training, and scoring. To deploy a trained model for realtime inference, I want to see if it's a good idea to reuse data processing and scoring pipelines. To minimize the latency, what's the best way to utilize nodes and pipes written in Kedro?

Gauthier Pierard

11/26/2025, 10:03 AM

hey, just to confirm there is no

AbstractDataset

predefined currently for polars to delta table? would something like this do the job?

Copy code

class PolarsDeltaDataset(AbstractDataset):
    def __init__(self, filepath: str, mode: str = "append"):
        self.filepath = filepath
        self.mode = mode

    def _load(self) -> pl.DataFrame:
        return pl.read_delta(self.filepath)

    def _save(self, data: pl.DataFrame) -> None:
        write_deltalake(
            self.filepath,
            data,
            mode=self.mode
        )

    def _describe(self):
        return dict(
            filepath=self.filepath,
            mode=self.mode
        )

Martin van Hensbergen

11/27/2025, 10:56 AM

Hello all. I am new here and am investigating whether our company should use Kedro. We work in a highly regulated industry where we need to trian and deploy ML models in a sound, versioned, reproducable way. Kedro seems to tick a lot of boxes when it comes to clear directory structure, Node/Pipeline concepts, tagging for partial execution, datacatalog, etc. I have a succesfully built a succesful kedro package with 3 pipelines: 1) preprocesisng, 2) model training and 3) inference. (1) and (2) work perfectly but it seems that at the realtime inference I have issues. After having batch trained ML models on large datasets, I need to use the trained models for per-point inference in a hosted service. So, training is batch wise but inference is point wise. It seems that you somehow need to do this by defining the inference input as MemoryDataset and then somehow load data in that and execute the inference pipeline. However, i cant seem to find a way to do this properly via KedrioSession. I wonder if the use case that I have is actually something that is supported out of the box for Kedro? Any advice on how to do this with minimal overhead? As for now I have defined input as

MemoryDataset

as input for the inference pipeline but I get "`DatasetError: Data for MemoryDataset has not been saved`" error when running:

Copy code

with KedroSession.create() as session:
    context = session.load_context()
    context.catalog.get("input").save("mydata")
    session.run(pipeline_name="inference")

1. Is this the proper way to do it? 2. Is this a use case that is supported by Kedro or should I only use it for the batch training and use the output of those models manually in my service.

Zubin Roy

11/28/2025, 12:04 PM

Hi all 👋 A quick question about Kedro versioning behaviour. Is it possible to do folder-level versioning rather than dataset-level versioning? My use case: I have a single node that outputs a dozen CSV files each week. We want to keep weekly snapshots, but downloading each versioned dataset individually is a bit painful ideally we’d like all files stored under a single timestamped folder. And also to me that's a much cleaner foldering way of storing the files and understanding the weekly snapshot. At the moment I’ve implemented this by generating a timestamp myself and returning a dictionary of partition keys, e.g.:

Copy code

timestamp = datetime.utcnow().strftime("%Y-%m-%dT%H-%M-%S")

return {
    f"{timestamp}/national_ftds_ftus_ratio_df": national_ftds_ftus_ratio_df,
    f"{timestamp}/future_ftds_predictions_by_month_df": future_ftds_predictions_by_month_df,
    ...
}

And my catalog entry is:

Copy code

forecast_outputs:
  type: partitions.PartitionedDataset
  dataset: pandas.CSVDataset
  path: s3://.../forecast/
  filename_suffix: ".csv"

This works, but I’m not sure if I’m using

PartitionedDataset

in the most “Kedro-native” way or if there’s a better supported pattern for grouping multiple outputs under a single version. It’s a minor problem, but I’d love to hear any thoughts, best practices, or alternative approaches. Thanks!

Lívia Pimentel

12/02/2025, 12:08 AM

Hi, everyone! I'm trying to parametrize a folder name via

--params

at runtime, but Kedro isn’t picking it up. In my

parameters.yml

I have:

Copy code

data_ingestion:
  queries:
    queries_folder: "${runtime_params:folder}"

Then, in the pipeline creation:

Copy code

conf_path = str(settings.CONF_SOURCE)
conf_loader = OmegaConfigLoader(conf_source=conf_path)
params = conf_loader["parameters"]

queries_folder = params["data_ingestion"]["queries"]["queries_folder"]

query_files = [f for f in os.listdir(queries_folder) if f.endswith(".sql")]

When I run:

kedro run -p data_ingestion_s3 --params=folder=custom_folder

I get an error saying "folder " not found, and no default value provided.
Has anyone used runtime parameters inside parameter files like this? Do you know if this is expected, or should I be loading params differently? I would appreciate any guidance you could give me! Thanks 🙂 Note: I am using kedro version 1.0.0

Jon Cohen

12/02/2025, 1:35 AM

Hi! I'm starting a new project up and my business partner wants to use kedro for it. I'm not super happy about being forced to use pip as opposed to something like pixi or poetry. Is there an official way to use a different package manager?

NAYAN JAIN

12/02/2025, 3:09 PM

Hello All! Is there any workaround for using

kedro viz

kedro viz build

when your catalog expects runtime parameters? I am not able to use these commands without manually deleting the catalog files. Are there any plans to support

--conf-source

--params

in the kedro viz command?

👀 1

Matthias Roels

12/02/2025, 4:39 PM

What's the current view on the future of using omegaconf? It seems the project is currently in “keeping the lights on” mode cfr https://github.com/omry/omegaconf/issues/1200

👍 1

Anna-Lea

12/03/2025, 2:25 PM

Hi Team 👋 In our workflow it happens quite often that we have a situation where: a node takes a

PartitionedDataset

as input and

PartitionDataset

as output. So something like this:

Copy code

def my_node(inputs: dict[str, Callable[[], Any]]) -> dict[str, Any]:
    results = {}
    for key, value in inputs.items():
        response = my_function(value())
        results[key] = response
    return results

Ideally, I would want: • the internal

for

loop to run in parallel I've noticed that @Guillaume Tauzin mentioned a similar situation

olver-dev

12/07/2025, 12:30 PM

@olver-dev has left the channel

Ralf Kowatsch

12/08/2025, 4:07 PM

Hi, sorry for the question but it happend multiple times that links that were referenced like docs.kedro.org/en/1.1.1/nodes_and_pipelines/run_a_pipeline.html#load-and-save-asynchr were dead. Is there some kind of issue I'm not aware of?

marrrcin

12/08/2025, 8:17 PM

Hi guys, long time no see! 😊 What would be current recommended way to do fan-out/fan-in processing in Kedro? Let's assume I would like to use either ParallelRunner or ThreadRunner to achieve some level of concurrency in the fan-out part.

👀 2

Marcus Warnerfjord

12/09/2025, 11:43 AM

Hi guys! I'm a bit new to the data engineering that kedro is bulding on, have limited experience in database management, and have a use-case that I'm struggling to find any straight-forward solution to. My pipeline is in essence creating datasets of DNA-sequences, which I send to an API (AlphaGenome) for property predictions, and then preform data analysis on. All of these sequence and predictions also needs to be stored in a global database for later use. The quirk of this workflow is that the API is enforces rate limiting, which not only motivates the need to only request predictions for new sequences (as predictions take a lot of time to produce), but also to have a failsafe if the API terminates the request, so that I can at least store the predictions I've received before that point. Additionally, the amount of sequences are substantial and this creates strict requirements on efficency in memory and speed. The consequences of this is that I always first need to check what's in the database before updating it, creating a circular dependency. I also need some sort of batch logic for updating my database to store progress. And because of this being a big database I can't really have any version logic that duplicates data. I know that one can create workaround by having two different dataset share the same dataset path but I have a feeling that there might be other design practices that are more aligned with kedros. Either way, I'm guessing that there at least exist a lot of similar use-cases that perform database updating who might have solutions to this. Thankful for any help, and if you have additional recommendation on good kedro-datasets for this, even better! :)

👀 1

👍 1

Julien Berman

12/10/2025, 8:29 PM

Hi all, I am an economics researcher interested in incorporating Kedro into our lab workflow. Our research projects typically involve ingesting and processing a large number of disparate datasets, so Kedro will help keep track of our pipelines. We are currently using SCons to create our task DAGs, which does not work well for large datasets. However, before introducing Kedro, I have a few questions. 1. We ingest, process, join, and aggregate all our data in Python. However, often, our analysis relies on R and/or STATA packages to produce tables and figures. Is it possible to integrate R / STATA scripts into Kedro pipelines? If so, how? 2. We currently have a set of scripts that validate data quality and produce log files / data summaries every time we write data to disk. Is there a way to somehow override the wrapper that Kedro uses to save data such that it will run these scripts? Julien

👀 1

Jordan Barlow

12/11/2025, 10:44 AM

Hello all, Has anyone managed to integrate a Kedro pipeline with Gitlab's MLflow client using the kedro-mlflow plugin? I always receive a

RestException: INTERNAL_ERROR: 401 Unauthorized

error despite having a properly scoped token. I have raised an issue in the kedro-mlflow repo: https://github.com/Galileo-Galilei/kedro-mlflow/issues/681 but thought to also ask here in case someone has already done this successfully, or I'm missing something obvious. Thanks!

👀 1

Shu-Chun Wu

12/11/2025, 3:56 PM

Hi all, I would like to disable some pipelines or tags or (some runs) from MLFLOW tracking. I tried to add below into mlflow.yml file, but if I still see it tracked on MLFLOW. On the other hand, how can I make sure the running sequence of pipelines? If I run like this, it wouldn't follow the order I wrote.

Copy code

run:
  tags: r_data_processing, r_regression_training, r_local_predict

disable_tracking: pipelines: - segmentation_evaluate - segmentation_preprocessing

Frank Wilson

12/13/2025, 2:02 PM

I'm trying to run the kedro viz backend in the vscode debugger. The backend process seems to exit with a non-zero code with no error message, after a delay of about a minute. Any idea what causes this? It seems this only occurs after control is passed to uvicorn but that's all I've managed to figure out.

Júlio Resende

12/17/2025, 4:55 PM

Hi! Does anyone has any material to share about best ways to integrate Kedro pipelines and Azure Machine Learning Pipelines? I took a look at the kedro-azureml
plugin, but unfortunately it depends on kedro 0.18.* 🙁

NAYAN JAIN

12/17/2025, 6:22 PM

Hello All! I wanted to implement a credential resolver function similar to this one: https://github.com/kedro-org/kedro/discussions/4320#discussioncomment-12141508 However, I do not want to put the conf_source derivation logic and the catalog read logic in here like @datajoely has put in

KedroCredentialResolver.get_credentials

, as my catalogs can be present anywhere (local system / s3) depending on scenario. Is there any way to access some kind of pre-loaded config or catalog object so that I do not need to look for and read the credentials files myself, and instead I can just rely on the pre-loaded conf object to simply resolve the credentials? I would want my catalog to look just the same as in the comment. catalog.yml

Copy code

my_dataset:
   type: ... 
   password: "${creds: snowflake_credentials, password}"

and creds.yml

Copy code

snowflake_credentials:
   password: "<secretsvault://secret-name>"

and my resolver would be able to use the secretvault url provided in

creds.yml

to fetch the secret whenever required inside the dataset. Is there any way to achieve something close to this? The reason for my interest in this implementation is that we have custom ways of getting credentials depending on the different scenarios and we want to make it very easy for someone to refer to them. We are okay with having to use custom implemented datasets to get this to work. However, having the above mentioned implementation combined with custom datasets could probably give the cleanest user-side catalog. Older question along similar lines: https://kedro-org.slack.com/archives/C03RKP2LW64/p1761746217970269

Merel

12/19/2025, 10:37 AM

<!here> Dear Kedro community 👋, We wanted to let you know that the Kedro Wizard wizard hat will be taking a well-deserved holiday break soon. Each week, a different member of the Kedro team steps into the Wizard role to help answer your questions, debug issues, and brainstorm feature ideas. Over the next two weeks, much of the team will be on holiday, so response times may be a little slower than usual. We really appreciate your patience and understanding 🙏 Thank you for all your support, thoughtful questions, and contributions to Kedro over the past year. It truly means a lot to us 💛 On behalf of the entire Kedro team and the TSC, we wish you happy holidays 🎄✨ and a wonderful start to the new year! Warm regards, The Kedro Team xmas kedro

K 5

party wizard 3

Emilio Vega

12/23/2025, 1:18 AM

Hi, for some reason i cant name a dataset type

spark.SparkDatasetV2

, it has to be

kedro_datasets.spark.SparkDatasetV2

. Im playing with 1.1.1, and databricks-connect Just to know if something has change, bc the docs say its the first way.