Hi team, There are 5 tables in my pinot cluster, I ran batch ingestion to pinot using spark jobs to these tables . The deepstore is an s3 bucket , the issue is after i ran the ingestion jobs, table A segments are getting pushed into table B deepstore folder, their schemas are different aswell. But I have verified in pinot controller UI table A segments are visible in the table B segments, so somehow controller is even registering these and servers are reading these into table B . I have confidence that my execution spec files are correct and verified. So please let me know of any plausible information regarding this.

What all did you validate to build that confidence? The segment is pushed to a table which it contains in the metadata, or specified in the push call.

i have verified the master node spark job logs and execution spec file . I had date wise partitioning , so verified on one of the particular mismatched date.

How do you specify which table a segment should go to?

The problem is I was using a shared staging directory and clean up was not happening regularly from spark side itself , i just got to know by reading some more logs of steps .

My point is segments contain table name in metadata, which comes from spec. The only way a segment S goes to a table T is because you species table T in the spec, or you are specifying in the push api.

I have specified correct table names in tableSpec field in executionspec file , if your reasoning is correct then there should not be a problem even if there is a shared staging directory .

I don’t see how staging directory being shared could cause that.