So I’d need to “hack” that to only show the ones that have actually materialized.

the unloading could also be done by the same periodic task which is responsible for moving from tier to tier (assuming you are planning to use the tiered storage design). Based on some flag in the tier config, for storageType=deepStore

will help in reducing 1 component

lets do a zoom call?

I’m down. When?

3 pm pst?

Earlier might be better, if you’re available

1:30 pm pst

Works for me

This message was deleted.

Might be a little more difficult than I had originally imagined. There’s really two entry points to downloading a segment,

SegmentFetcherAndLoader

and

RealtimeTableDataManager

. Unifying those two seems like it may be difficult, as the realtime use case has some backup peer downloading.

The peer downloading should be applicable to both offline and realtime (might not be the case right now)

And all segment download should be handled within the same class

Yeah, appears it is not handled that way now. The only way it knows the uri for the deep store download is from a realtime specific metadata class

This bit of the codebase could use a refactor, but I’m not sure I have the time

Neat, got the lazy loading working. Going to add some metrics and clean this up a bit.

Ah, also need to make it download direct from s3 instead of the controller. That's gonna put load on the controller for a big query

That’s already supported

It depends on how segment is pushed

If you use uri based push, it automatically downloads it from S3

@User you can try to use

jobType: SegmentCreationAndMetadataPush

in the spec.yaml file

if your output directory is already s3

# outputDirURI: Root directory of output segments, expected to have scheme configured in PinotFS.
outputDirURI: 's3://<your-bucket>/pinot/<table>/segments'

I think server download from s3 is only done for real time tables.

it’s both 🙂

you also need to configure pinot fs on server as well

this tutorial uses segment uri push, since you already on newest version, segment metadata push will be much better

Yep, can verify with the SegmentCreationAndMetadataPush there’s no data that goes through the controller, downloads straight from s3. Of course,

select * limit 1

causes it to download all of the segments lol. Really going to need that time based segment pruner. Also will need an option to limit the number of segments a query can hit, made an issue here

https://github.com/apache/incubator-pinot/pull/6250 The PR for this feature Dunno if it’s just the permissions, or github’s user interface has changed for the worst since I last used it --but I can’t figure out how to add reviewers or labels.

I think only committers can do that