which seems fine - but the implication is we're kinda duplicating storage for the offline segments. Eg: in our setup - each table owner has her own namespace in HDFS where the Pinot segments are initially created. After the segments are pushed, the segments are effectively copied from the customer's namespace to Pinot's namespace. Is there a possibility we can retain the behavior of using the original download URI ? This will make our chargeback / quota management story very easy

@User ^^

@User We support both. In

PinotSegmentUploadRestletResource.uploadSegment()

, when

moveSegmentToFinalLocation

is set to false (upload segment with json), we are using the original download URI

yes, but I do see a comment saying its going to be deprecated. I guess I was hoping we don't do that

Fix for the download URL

We have run Pinot on Kube.

@User recently added instructions here https://github.com/apache/incubator-pinot/tree/master/kubernetes/examples/gke/skaffold

<!here> Mayank, Seunghyun, and Jackie will talk about building real-time analytics applications using Pinot at Apache Con. Venue El Dorado at 3:30.

Please share it in your network

👍

link ?

One sec

ApacheCon Pinot talk slides

is this ThirdEye or something else?

That’s Raptor

We plan to open source it shortly

cool!

Hey guys, we're running into this exception in Pinot-controller. Any ideas why and if this is harmful?

Caught exception in segmentCommitEnd for segment testEventView__1__18__20190911T2002Z
java.lang.IllegalStateException: null
	at com.google.common.base.Preconditions.checkState(Preconditions.java:429) ~[guava-20.0.jar:?]
	at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:432) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-235a51f841975d943e874a1358e50e2ab55755fc]
	at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:396) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-235a51f841975d943e874a1358e50e2ab55755fc]
	at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.commitSegment(SegmentCompletionManager.java:1035) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-235a51f841975d943e874a1358e50e2ab55755fc]
	at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.segmentCommitEnd(SegmentCompletionManager.java:634) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-235a51f841975d943e874a1358e50e2ab55755fc]
	at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentCommitEnd(SegmentCompletionManager.java:309) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-235a51f841975d943e874a1358e50e2ab55755fc]
	at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommit(LLCSegmentCompletionHandlers.java:322) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-235a51f841975d943e874a1358e50e2ab55755fc]
	at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.28.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.28.jar:?]

how does your ideal state look? does it contain

testEventView__1__18__20190911T2002Z

segment in

CONSUMING

state?

Created channel #CNBLA2M6Y if you are interested

In case you folks haven't seen this already: http://www.vldb.org/pvldb/vol12/p2059-zhan.pdf

main idea is to separate write nodes and read nodes backed by a distributed storage (blob ? ) system . Also supports joins, upserts/deletes

good read !

Can Pinot currently serve a use case where: - the data stream has low density per-key within a specific time window (1-5 events per key within time window) - the data stream has very high cardinality of keys (10^10 or 10^11) - query latencies must be <40ms for extremely simple aggregation queries? (

select count(*) where key='a'

,

select min(longTypedColumn) where key='a'

)

Yes. Configure key as sortedKey

Thank you.

Is the answer still yes if I clarify that the stream being queried is ~10^7 records/second, with ~10^4-10^5 queries per second?

or do those parameters fall under Pinot's horizontal scaling parameters