I'll look into some way of parameterizing a notebook or script to either run in pandas locally or distribute jobs to a config'd spark cluster.

a similar thread: Can we specify the interpreter?

# src/ploomber/tasks/notebook.py
if self.source.language == 'python':
  interpreter = _python_bin()

use case: Connect to local interpreter (default) or remote interpreter. Databricks has a remote kernel integration that allows local notebooks to be executed on Databricks. This is probably also something that could be parameterized at the task level, but it'd be nice to toggle for an entire DAG. note: I'm (probably incorrectly) equivocating interpreter and kernel here, but I hope the intent is clear.

ok, so I read the article. looks like it's possible. you can customize this:

from pathlib import Path
from ploomber import DAG
from ploomber.tasks import NotebookRunner
from ploomber.products import File
dag = DAG()
NotebookRunner(Path('nb.ipynb'), File('report.html'), dag=dag, papermill_params=dict(kernel_name='kernel-name'))
dag.build()

to list the kernels available, you can run

jupyter kernelspec list

, then you can substitute

kernel-name

for the kernel you want to use. I don't have access to a databricks cluster, but let me know if this works