looks like this ^^ was a different issue. Still need help with the original problem. @Kishore G @Mayank ☝️

is this try to writing to a remote disk or local disk

I used to see issue of writing to remote disk

however that time the written succeed, but segment file is corrupted

local disk

Probably you already checked, but could disk space be the reason?

yes, there was enough space

Can you show the logs? There must be something overflowing.

which logs?

Only thing I could get was message printed on console which was exception stacktrace

nothing before that?

no

it should print the cardinality of each column and total number of records

no cardinality info

Could you share the data characteristics - number of files, number of records per file, cardinality,data type,field type,value range, max length for each column @kish

STRING column; Cardinality: 1; Max length bytes: 15 • STRING column; Cardinality: 8K; Max length bytes: 25 • STRING column: Cardinality: 38K: Max length bytes: 15 • STRING column: Cardinality: 4; Max length bytes: 4 • STRING column: Cardinality: 35; Max length bytes: 26 • STRING column: Cardinality: 3; Max length bytes: 10 • STRING column: Cardinality: 5; Max length bytes: 8 • STRING column: Cardinality 5; Max length bytes: 5 • STRING column: Cardinality 157; Max length bytes: 29 • INT column: cardinality 3; range 1 - 50 • INT column: cardinality 63971; range 0-64K • LONG column: cardinality: 3.5 million - timestamps

1 file, 3.5 million records

This is the same as before? This had worked fine when creating segment from a CSV file right? @kish

yes

Hello, at the moment i am using the quickstart.java to run Pinot, i am doing my tests in a table with 30M Of documents, only two dimensions (id, group_id) and one metric (amount) how can i understand "when" to scale Pinot on more servers?

i think the quickstart.java is not tuned, i mean we can do more to optimize the latency i suppose

do we have trim transformation in pinot

trim expression ?

No, but should be easy to add. What syntax semantics are you looking for?

trim the whitespace

transformation during ingestion or query?

In superset , column expression? while ingestion , hope groovy will support that. Actuall i already ingested and need to do a quick demo.

I dont think we have one for String functions

you need something like this for String