sqlmesh python models + daft would be 🔥 https://sqlmesh.readthedocs.io/en/latest/concepts/models/python_models/#pyspark

I haven’t heard of this library before. What sort of things do you use it for?

they describe it as a "data transformation framework" that is supposed to remove some of the boilerplate around setting up things like batch incremental ingestion, dependencies between data transformations (e.g. table 1 depends on table 2), testing/auditing data transformations