Hi folks, I'm planning to do a PoC to evaluate Daft for distributed multimodal data processing and would love pointers from the community.

POC scope:

- Multimodal pipelines (unstructured data + image/video + Audio) for AI workloads — embedding generation + inference - Heterogeneous compute: CPU for I/O and preprocessing, GPU for model inference - Reads/writes against object storage/vectordb (Parquet/Iceberg/raw files) - Distributed UDFs (including GPU-bound ones) at scale - Autoscaling behavior under bursty workloads

What I'm looking for:

1. Is there a recommended cookbook / reference setup for standing up a distributed Daft cluster (Ray-based I assume)? Any opinionated deployment guides for K8s? 2. Best practices for mixed CPU/GPU worker pools: resource tagging, fractional GPU allocation per UDF, scheduling hints. 3. Patterns for embedding + inference pipelines: batching, model loading per worker, keeping GPUs saturated. 4. Autoscaling story: what works today (KubeRay autoscaler?), known gotchas. 5. Recommended read/write connectors for multimodal data at scale (images/video blobs + metadata). 6. Any PoC templates, example notebooks, or reference architectures the team points users to first. Thanks in advance!

This is a nice starting point for some examples, https://github.com/Eventual-Inc/daft-examples

additionally https://github.com/Eventual-Inc/Daft/tree/main/benchmarking/ai has some examples of how we usually set up ray cluster.

Thank you will go through it!

hope this helps

I am looking for running on Kubernetes (via kuberay), but thanks!