Quick question on multi-controller behavior in Pinot: From the /leader/tables API, it looks like tables are distributed across controllers, and schema changes (e.g., adding a column) go through the table’s leader. How frequently do controllers sync metadata? Scenario: I added a column to a table whose leader is c2, but when querying via c1 UI, I temporarily don’t see the column (especially in multi-stage engine setups where different tables have different leaders). Restarting c1 resolves it. Is this expected behavior, and are there configs or best practices to avoid this lag/glitch? @Mayank @Xiang Fu

That is not expected. Schema is stored in ZK, not in controller, so what you are describing isn't really related to lead controller, I suspect something else might be going on

Is there a way I can debug this further? The exact error while querying the join was “Error Code: 700 QueryValidationError: From line 40, column 3 to line 40, column 12: The definition of column ‘uuid’ depends on itself through the following columns: ‘uuid’ The same query executed correctly on c2 controller, but on c1 UI looks like it failed during validation and hence the query never reached to broker. Instead of join, if I just select uuid from pinottest table this also works.

You have multiple controllers. Are you using deepstore?

Yes, s3 is our back up store

Once you change the schema, did you reload all segments?

Yes, I did. It was quite surprising for me as one of the controller with Multistage enabled join query is working but not on another. I assumed it might be the new schema didn't get sync correctly to the first controller, but you have confirmed its not the case.

Schema is stored in ZK, so there is nothing to sync across controllers.

I wonder how restarting the controller resolved the issue.

Hard to debug over slack. May be it is SSE vs MSE behavior and after restart both controllers run MSE?

Can understand. Okay, but doesn't SSE throws different type of error when you try to query with Joins? It explicitly says to use MSE. Have you experienced or any one from your team the error I posted?

If you can consistently reproduce, may be file a bug, with repro steps?

Sure, let me see if I can reproduce again. Thanks Mayank 🙏