Hey! When I run spark ingestion job on local system, I can see that pinot-plugins-dir-x (where x is int) directiories are being created. What is it? Where would it be created if I ran spark-submit job on k8s? Would it create on executor pod?

@Kartik Khare do you know the answer?

Hi, they are created in spark's staging dir. You can set that in the conf. That being said there was an effort in progress to not create these dirs. I need to take a look at it again.