are there any practical limitations on generator udfs (e.g. 1 row to 100k rows) or should this scale?

As of right now, the generator UDF is fully materialized. So if you have N input rows and each yields M output rows, the entire N×M result is collected in memory before being expanded (similar to an explode). For moderate fan-outs like 1 → 100k this should work fine, but you'll want to be mindful of memory as N×M grows large (e.g., 10k input rows each yielding 100k rows). We do plan to make this truly lazy in the future, which would allow streaming results without holding everything in memory at once.

into_batches

will be the tuning parameter here for N, right?

yes, using smaller batch sizes would mean smaller memory footprint. We are also working on auto tuning of batch sizes for explodes: https://github.com/Eventual-Inc/Daft/pull/6069