https://kedro.org/ logo
Title
a

Andreas_Kokolantonakis

05/25/2023, 1:23 PM
hello everyone! does anyone have an example using partition by when saving parquet files using kedro catalog? thank you very much in advance
n

Nok Lam Chan

05/25/2023, 1:25 PM
a

Andreas_Kokolantonakis

05/25/2023, 1:28 PM
hmm, I can see this issue is still open, I would like to specify the partition columns to save in the save_args.
n

Nok Lam Chan

05/25/2023, 3:39 PM
What’s your existing config looks like? If you just need partition by columns you should be able to use SparkDataSet itself but not PartitionedDataSet
save_args: Save args passed to Spark DataFrame write options.
Similar to load_args this is dependent on the selected file
format. You can pass ``mode`` and ``partitionBy`` to specify
your overwrite mode and partitioning respectively. You can find
a list of options for each format in Spark DataFrame
https://docs.kedro.org/en/stable/kedro.extras.datasets.spark.SparkDataSet.html