Hi team, in Uber, we want to increase the Pinot's ...
# pinot-dev
j
Hi team, in Uber, we want to increase the Pinot's SQL parser identifier max length (i.e. table column name length) to more than the default 128 characters. This is can be done by setting the identifierMaxLength in the Calcite SqlParser config. The code which set up the configs in Pinot is in
src/main/java/org/apache/pinot/query/parser/ParserUtils.java
. Obviously, we can hard-code a larger max length which is fine with us. For OSS, is there any recommendation on the mechiansm to pass in this max length as a config property?
Another potential question is that can we simply increase the 128 char limit to something higher in the OSS? Are there any potential downsides?
@Sean Sullivan Any updates?
Just a quick update, the config in
ParserUtils.java
is actually dead code and the code that actually gets used is in
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
.
Specifically,
Copy code
static SqlParserImpl newSqlParser(StringReader inStream) {
    SqlParserImpl sqlParser = new SqlParserImpl(inStream);
    sqlParser.switchTo(SqlAbstractParserImpl.LexicalState.DQID);
    // TODO: convert to MySQL conformance once we retired most of the un-tested BABEL tokens
    sqlParser.setConformance(SqlConformanceEnum.BABEL);
    sqlParser.setTabSize(1);
    sqlParser.setQuotedCasing(Casing.UNCHANGED);
    sqlParser.setUnquotedCasing(Casing.UNCHANGED);
    sqlParser.setIdentifierMaxLength(<some configurable value>);   <---- can add config here.
    return sqlParser;
  }
If we want to make it configurable, adding a static variable, and setting it in some configuration parsing place could work. So any recommendation on a good place to specify the max identifier length should be a sufficient guidance.