Hi Team, I need few details, as we are working to onboard few more usecases to Pinot . https://github.com/apache/pinot/releases/tag/release-1.0.0 1. Does pinot support direct spark writes , specifically dataframe persist to Pinot as offline table. what i know is spark requires dataframe to persist (parquet), then load to pinot a two layer job. Any improvement for direct write to Pinot? https://docs.pinot.apache.org/users/tutorials/ingest-parquet-files-from-s3-using-spark 2. Is there any more documentation on null handling, specifically with enable_null_handling=true. https://github.com/pinot-contrib/pinot-docs/blob/latest/developers/advanced/null-value-support.md I do see specific document for null handling, but it says handling null in aggregation is not supported .Would like to confirm is this behaviour changed in latest release or remain same.

1. No, Pinot does not support direct spark writes. What is the use case? 2. Yes Null support is in progress and isn’t available for aggregations yet

Thanks , On spark use-case , our customer want one less ETL to monitor to track of load into Pinot and avoid intermediate persist. As now it has to be stored on hdfs/s3 and then again loaded.

That Pinot jar requires physical files in certain formats to be available. It should be possible to convert it directly from data frame if that’s what you mean (but doesn’t exist today). @Jack I remember you did something like this in the past in closed source?

Thanks for the prompt response. Following on the same lines. Currently, how this physical files will be read in the pinot-spark loader?.