Let’s say datetime

And user can specify granularity at query time and Pinot picks the right physical column

This is auto

Vs user query using the physical column datetime-15m ...

Ok I get it now, so I could have multiple physical columns which are rollups? The query time determines the best fit? Based on hints?

Yeah. That was the idea but It’s not implemented yet.

So for now, user needs to use the physical column explicitly

ok

Does the

dimensionFieldSpecs

support enums?

It does not matter to Pinot

Does that mean its not enforced at the pinot level but data ingested into pinot should have those constraints pre-applied?

Does pinot apply the downsampling before ingest or after?

Yes for enums

Pinot does not sample

And for rollups, are they pre or post ingestion?

post ingestion

it can also aggregate while ingesting. It really depends on how many rollups do you want to maintain

if multiple, then use star tree

fyi: is there anyone who is working for linked in? If yes, funny bug for you: posting a flink error into messaging in Firefox completely killed it. Can’t message anyone anymore. The killer:

You: 2019-11-12 22:10:23,411 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [/10.72.5.85:43946] failed with <http://org.apache.flink.shaded.akka.org|org.apache.flink.shaded.akka.org>.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 10485760: 1195725860 - discarded

A lot of folks here work at LinkedIn. Thats the error message you got?

yep. Try sending it to me for example 🙂

dont have firefox

hey, question about pinot-servers. Kube pods have port 8098 opened. Is it a REST endpoint? Seeing a weird thing -> I can query the cluster using Pinot UI. netcat works with pinot-server, but curling it times out, even on localhost.

our presto connector also times out talking to servers

brokers are fine

servers dont have rest end point

they have thrift end points

ok, got it

did anyone try to generate pinot segments using Spark or Flink (preferably Flink), or even Beam?

Spark - yes. Flink - no beam - no