For me,

controller.port

is a bit of a sneaky one. It is a transitive property. Breaking it down would make its effect and behaviour more predicatable.

It could be used in differents contexts accross different version. and is inolved in your identity on helix without being obvious.

let's get the PR out and look at the actual variable names. With more deployments moving to cloud (k8s), there is some clean up needed on how we compute the instance id to connect to Helix. But lets take that up in a separate PR

That’s a WIP : https://github.com/apache/incubator-pinot/pull/5543

cool, lets continue our discussion on the PR

I think that there are two different points that we need to think of. 1. minimize the user impact using canary, early detect the issue by improving the testing (e.g. setting up perf cluster with representative production use cases): I think that we all agree on this point. @LinkedIn, we already have the canary mechanism and we also do have a plan to further improve our testing pipeline for internal use cases to be able to detect the issue much earlier. 2. pinot community’s stance on backward incompatible changes: As Pinot gets mature, we should definitely be careful about the backward compatibility because this will increase the complexity for users when they upgrade the versions. I do agree that breaking backward compatibility should be evaluated based on the cost of upgrade vs benefit that the change will bring into, not based on the risk of LinkedIn deployment. As long as the backward incompatible changes are correctly applied (keep both behaviors -> kill the old behavior later), we do not have any problem with it. But many times, we’ve had many production issues because some commits broke the backward compatibility within the same version, which is not the expected behavior. This is something that we as a community should put the effort to improve.

@User and @ Kenny are approaching from new user/operator side and Subbu and others are approaching from existing deployment perspective.

Both are valid and I think as most of you have already pointed, looks like we can achieve both in this case.

Please vote for enabling html feature for pinot mailing lists. This is needed for setting up the daily digest mails from slack discussion.

I sent out the mail to dev@pinot mailing list

Hello, I shared the design proposal for Filtering during ingestion on dev list. Would really appreciate feedback/suggestions https://docs.google.com/document/d/1Cahnas3nh0XErETH0KHLaecN6xCnRVYWNKO3rDn7qcI/edit#

@User will be good to send out an email as well, with a summary of proposed config changes. With pinot gaining wider adoption in the community and more production cases, we need to pay attention to our configs and make sure we pick the right defaults and keep things as simple as possible. The community can help us here

I've submitted a PR to remove the code for HAVING clause handling because it is not properly done and the result is not accurate: https://github.com/apache/incubator-pinot/pull/5570

Officially (in the documentation) Pinot does not support HAVING for now. Will add HAVING clause support after migrating to SQL

I’m trying to move pinot startable to use config files always: https://github.com/apache/incubator-pinot/pull/5446

this can better fill the cap between trying out our quickstart and deploying pinot

Consistency is awesome! Just a quick question on the theme around this improvement. Has it been considered to standardize config management so all the potential properties may be passed through env var, cli args and property files independently? Most modern java framework offers configuration abstraction. Apache Commons Configuration enables that with CompositePropertiesConfiguration but it requires manual setup and bootstrap.

The purpose is to let startable not using the cli args but use config file

so it would be better place to enforce the consistency

I understand the purpose of the PR, and think is a good improvement. My question is wether it has been considered to implement an abstraction that offer the maximum flexibility for operators. Based on the deployment method, cli args might be the best. In the case of K8s, env var are more likely to be preferable, and on template engine based deployment, config files are great. I like to think that pinot should be independent from the configuration source.

This question is off topic from the PR, I’m forecasting ideas for future improvement for maximum flexibility for operators.

hmm

let me think about it

ideally there could be templating configs

then config override from cli is supported

I feel most of the deployment should be fine with updating the config file and restart

@User we want to standardize the start/stop of all components

will create an issue and share my thoughts

Config management is a topic of the heart, by such it is hard to assume what people prefer. By experience, what usually finds appeal in a community is choice. The best configuration mechanism is tied to a context which that cannot be assumed by a software contributor. Long story short, abstract it from your software and everyone will be happy. Yaml, json, env var, java system properties, implementation detail and not much of a deployment constraint.