Thanks!

in case you need references: https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers

Do I just delete the tag or do I edit it? Here is the contents:

get /pinot/CONFIGS/PARTICIPANT/Server_pinot-server-5.pinot-server-headless.pinot.svc.cluster.local_8098
{
  "id" : "Server_pinot-server-5.pinot-server-headless.pinot.svc.cluster.local_8098",
  "simpleFields" : {
    "HELIX_ENABLED" : "true",
    "HELIX_ENABLED_TIMESTAMP" : "1596492023697",
    "HELIX_HOST" : "pinot-server-5.pinot-server-headless.pinot.svc.cluster.local",
    "HELIX_PORT" : "8098",
    "adminPort" : "8097",
    "shutdownInProgress" : "false"
  },
  "mapFields" : { },
  "listFields" : {
    "TAG_LIST" : [ "DefaultTenant_OFFLINE", "DefaultTenant_REALTIME" ]
  }
}

Is there a recommendation on how wide a table can be? i.e. we have a ~110 column table. Would it be better to split it vertically into smaller sized tables?

If the individual queries are not spanning a lot of columns, it should be fine I think. However, with large dimensionality, you won’t get much pre aggregation (if you care about latency).

If you spilt, you might be able to normalize better, but will need a join

110 is fine, you can use star tree for important columns

I think we may have gone a little too overboard with the indexing: 30 dimensions in the dimension split order and ~280 function column pairs. Should we reduce that?

wow, thats definitely pushing it a bit

is it one star tree or multiple trees

Just one

How many columns would you recommend for the dimension split order?

lets start with the issue you are seeing

is there any issue with this configuration?

Not seeing errors, right now it just looks like the offline segment we are generating is ~35gb with all the indexes.

And it fails to upload

Segments should be ~200mb, right?

got it, yes

whats the startree threshold

maxleaf threshold? It's the default

10k

Ok

How many rows in segment - Raw size?

221k - funny, I removed the star tree indexes and the segment directory is only 73mb.

Interesting. How’s the latency without star tree?

You are not getting much benefit from star tree for such small segments

@Cesar How many CPU cores?

Also, could you provide a bit more context on what is the goal here?

16 Cores. 16GB RAM.