Ploomber is more like an orchestrator than a distributed computing solution, but if you're doing simple stuff like mapping, you can implement it with the grid feature:
https://docs.ploomber.io/en/latest/cookbook/grid.html
essentially, you could define a grid of parameters and each of those parameters could be the indexes to process e.g., first task processes the first 1 M rows, the next one another 1M and so on. parquet is a a great format for these type of things
however, for more complex operations, I'd recommend you go with a distributed computing framework